I will present the current research activities in our Image and Video Understanding Laboratory at KAUST. The applications will range from video understanding to autonomous navigation, with solid fundamentals in deep learning theory. First, I will introduce the IVUL research on Video Understanding, including pioneer work on activity proposals and localization, video retrieval and visual object tracking. Then, I will present our works on Vision for Automation, focusing on 3D computer vision and imitation learning applied on self navigation, underlining the importance of simulation and transfer learning. Last, I will present the on-going research on Deep Learning Fundamentals, including large scale optimization, analytical expression for neural networks and robustness to adversarial attack.