We seek the ability to take a few images of a scene of interest, and turn it into an immersive visual experience, where one can explore it from different viewpoints, in effect visualizing a 3D representation of an object, scene or photograph, and providing numerous applications in augmented reality, e-commerce and 3D photography. This problem, known as view synthesis or image-based rendering in computer vision and graphics, has a three-decade plus history, and is currently undergoing a renaissance with new representations of 3D geometry enabling unparalleled realism. We discuss some of the history in terms of capturing the light field (the space of light rays for any spatial position and viewing direction), and our own work on a sampling theory for view synthesis based on light fields, leading to the development of volumetric radiance fields as a fundamentally new approach to representing 3D geometry for view synthesis. We will also discuss parallels to Monte Carlo and volumetric rendering and simulation problems in computer graphics. We then ask the question of how far we can push the required number of images, in order to achieve sparse view synthesis with very few images, in the limit only one photograph. In this context, we also discuss our recent results on a number of applications including real-time live portraits, generative AI for 3D scenes, and differentiable light transport for inverse rendering.

Ravi Ramamoorthi

Ravi Ramamoorthi is the Ronald L. Graham Professor of Computer Science at UCSD and founding director of the UC San Diego Center for Visual Computing. He earlier held tenured faculty positions at UC Berkeley and Columbia University, in all of which he played a key leadership role in building multi-faculty research groups recognized as leaders in computer vision and graphics. He has authored more than 200 refereed publications, including 100+ ACM SIGGRAPH/TOG papers. He has consulted with Pixar and startups in computational imaging, and currently holds a part-time appointment as a Distinguished Research Scientist at NVIDIA. Prof. Ramamoorthi has received more than twenty major honors including the ACM SIGGRAPH Significant New Researcher Award for his research in computer graphics, and the Presidential Early Career Award for Scientists and Engineers for his work on physics-based computer vision. He is a fellow of IEEE, ACM and the SIGGRAPH Academy, received an inaugural Frontiers of Science Award and again the following year, and has twice been honored with the edX Prize certificate for exceptional contributions in online teaching and learning. He has graduated more than 30 postdoctoral and Ph.D. students, whose theses have been recognized by the ACM Dissertation Award honorable mention, the ACM SIGGRAPH outstanding dissertation award and the UCSD Chancellor's Dissertation Medal.