Amitabh Varshney

Immersive Reality Beyond Pixels: A Journey into Precision Virtual Environments

Amitabh Varshney, University of Maryland

Abstract

The demand for high-precision virtual environments is burgeoning across diverse fields such as education, simulation, training, performance, and entertainment. In this presentation, I will introduce HoloCamera, an innovative volumetric capture instrument designed for the swift acquisition, processing, and creation of cinematic-quality virtual avatars and scenarios. The HoloCamera, featuring a custom-designed free-standing structure, incorporates 300 high-resolution RGB cameras strategically mounted with uniform spacing on the four sides and ceiling of a room-sized studio. The acquired light field undergoes sophisticated processing through a distributed array of GPUs, interleaving the handling and transmission of 4K resolution images, posing unique challenges in processing dense light fields.

To tackle these challenges, I will introduce neural methods, emphasizing their potential in representing light fields. I will specifically discuss our recent breakthroughs in employing a novel input transformation strategy based on Gegenbauer polynomials, known for their theoretical advantages over the Fourier basis. The resulting Gegenbauer-based design, coupled with sinusoidal activation functions, outperforms various network designs, including those incorporating Fourier-inspired techniques from previous works.

Furthermore, I will introduce a progressive multi-scale light field network that efficiently encodes varying levels of detail within a light field. This innovative approach allows for reduced rendering time through fewer neural network weights at lower detail levels, addressing aliasing concerns and supporting dithered transitions and foveated rendering.

In concluding the talk, I will discuss future of precision virtual environments and their wide-ranging applications.