A Cave System for Interactive Modeling of Global Illumination in Car Interior

1. A Cave System for Interactive Modeling of Global Illumination in Car Interior Kirill Dmitriev, Thomas Annen, Grzegorz Krawczyk, Rafal Mantiuk, Karol Myszkowski, and Hans-Peter Seidel Max-Planck-Institut für Informatik, Saarbrücken, Germany Karol Myszkowski

2. High Dynamic Range Human eye adjusts comfortably up to 12 orders of magnitude and can see simultaneously up to 4 orders of magnitude

3. HDR Pipeline: Acquisition • Global illumination • Products: HDR cameras • Lars III (Silicon Vision), Autobrite (SMal Camera Technologies), HDRC (IMS Chips), LM9628 (National), Digital Pixel System (Pixim) • Technology [Nayar 2003]: • Trading-off spatial resolution • Multiple sensor elements within pixel • Spatially varying pixel exposures • Mosaicing with spatially varying filter • Trading-off temporal resolution • Multi-exposure LDR capture • Multiple image detectors (beam splitters) • Smart pixels

4. HDR Pipeline: Storage • Several formats for still images • Radiance • OpenEXR (new industrial standard) • logLuv tiff • HDR JPEG • HDR MPEG

5. HDR Pipeline: Display • LDR displays: luminance compression required • Solution: tone mapping • Important factors in tone mapping selection • Dynamic range of display device • Video display conditions • Background lighting • Application • Just nice looking video • Visually plausible results • Optimizing visibility of details • Improving contrast … • HDR displays start to appear • Sunnybrook Technologies and University of British Columbia

6. HDR Pipeline: Applications IMS Chips

7. Realistic Image Synthesis Light Reflection Light Transport Simulation Visual Display Display Observer emission geometry BRDF displayed image radiometric values 27 15 22 13 HDR HDR LDR Greenberg et al. Siggraph’97. Cornell University

8. Acquisition: Materials • Shift variant Bi-directional Reflectance Distribution Function (BRDF) • Lensch et al. • Transluscency • Goesele et al.

9. Acquisition: Luminaires • Luminaire spatial power distribution (Goesele et al.) • Near field photometry • Emitted energy represented as 4D light field • Relevant for light sources installed in the car interior measurement VR: rendering

10. Acquisition: Luminaires • Natural lighting • Light probes and multi-exposure techniques • SpheroCam from Spheron VR • Emitted energy represented as an environment map • Relevant for external car illumination Light probe VR: rendering Paul Debevec

11. Lighting Simulation + Rendering 1) Photographs of mirror sphere at varying exposure times 3) Use as light source in Monte Carlo radiosity algorithm 2) High-dynamic range environment map Philippe Bekaert

12. Our Goal • CAVE system for car interior rendering • Real-time lighting simulation • Dynamic real world lighting • HDR video environment maps • Free observer position • Head tracking • Special focus: • Predict impact of quickly changing lighting conditions on the visibility of information displayed at the LCD panel • Precise modeling of light reflections from the LCD panel • Predicting effective contrast and information readability for various viewing angles • Taking into account light adaptation conditions for the human eye

13. Requirements for Rendering Algorithm • 5x2 full screen resolution frames at interactive rates • LCD panel illumination must be computed precisely • Higher error tolerance for the car interior illumination • Only low spatial frequencies in reconstructed lighting acceptable • Distant lighting assumption holds • HDR environment maps acceptable • Lighting can change quickly • Relying on temporal coherence between frames impractical • Car interior geometry static

14. Rendering Algorithm Selection • Requirements • Exploit all computational resources available in the CAVE: 20 CPUs + 10 GPUs • Rendering solution • Use Precomputed Radiance Transfer method for car interior: 10 GPUs + 10 CPUs • Use Final Gathering for the LCD panel: 10 CPUs

15. Spherical Harmonics • Spherical Harmonics: • Orthonormal basis over the sphere • Analogous to Fourier transform • Projection: • Reconstruction: • Integration:

16. Incident Light Visibility Cosine Rendering Equation ReflectedLight Jan Kautz

17. into SH into SH light function: transfer: Precomputed Radiance Transfer "light vector" "transfer vector" Such a dot product must be computed for each mesh vertex: n= 25 Jan Kautz

18. PRT for Arbitrary Meshes After Before

19. Handling Two-Sided Geometry • Car geometry is two sided • Need to store SH coefficients for both sides and render geometry twice with back face culling • This leads to lower performance and twice larger memory consumption • Better store SH coefficients only for one side of geometry: • Only the rays going through the windows contribute to SH coefficients • User has to point out the mesh parts representing car windows

20. LCD Panel Modeling • Emission characteristic of the LCD-sandwich: • Spectral emission-function of the backlight • Transmission characteristic of the polarizer and other layers (e.g. BEF, d-BEF, LCF, rgb-filters, …) • Transmission characteristic of the LC-cell • Transmission of other optical elements – e.g. glass • Reflective characteristic in case of incoming light • Surface coating (e.g. AR/AG) • Reflection characteristic of the LCD sandwich observer Incoming lighting glassplate LCD-sandwich backlight

21. Computing LCD Panel Lighting • DIMOS or SPECTER systems can be used for lighting simulation within the LCD panel • We use just external spectral emissivity and reflectance data provided with a very high angular resolution • Algorithm • Cover geometry of LCD panel with texture • For each texel • Compute energy emitted in the observer direction • Use tabulated emissivity data • Modulate emissivity as a function of the displayed information • Compute energy reflected in the observer direction • Use the final gathering method • Improve performance using BRDF-weighted importance sampling • If traced ray hits the car window, query the environment map • If traced ray hits the car interior, query PRT data structures

22. System Architecture

23. Distributing the Computation • Use “Lightning” system that takes care of OpenGL synchronization between different PCs • LCD display computation is distributed in an asynchronous way Camera parameters

24. Distributing the Computation • Use “Lightning” system that takes care of OpenGL synchronization between different PCs • LCD display computation is distributed in an asynchronous way Broadcast the display image as it is ready

25. Tone Mapping in the CAVE • Visual adaptation is influenced by light projected on a small area around the center of retina • Head tracking enables precise estimation of the gaze direction • We assume that the adaptation is affected by scene luminances within the region of 10° surrounding the gaze direction • This region can be mapped to 1-3 walls in the CAVE • Luminance data within this region is collected and send to the master • Master computes common tone mapping parameters and broadcasts them to all computers in the cluster • There is a small delay in illumination update, but it is hidden by temporal adaptation model anyway

26. Rendering with Tone Mapping • 3 Passes: • Compute tone mapping parameters • Preview rendering of 128x128 HDR images for the CPU processing • Select visible geometry for tone mapping • Lighting OFF • Render geometry to z-buffer • Final rendering with tone mapping • Lighting ON • Use z-buffer test to tone map only visible fragments

27. Results (Car Interior Part) • Car model contains approx. 500K triangles • Preprocessing of SH coefficients takes about 100 minutes • Full model size with SH coefficients is about 60 MB • Rendering frame rate is about 10 FPS • For each frame current environment map is projected into SH basis and lighting in every vertex is computed • Full image tone mapping is applied

28. Results (LCD Panel Part) • One processor on each PC is busy with sending data to OpenGL • Another processor computes draft images (40 samples per pixel) of LCD panel and sends them to other PCs • Draft image display is available almost immediately after each head movement or environment maps rotation, converged image is computed in approx. 2 seconds

29. Conclusions • We proposed efficient global illumination and tone mapping solutions for CAVE VR applications involving static geometry and dynamic environment lighting • We successfully applied those solutions to the car interior modeling, efficiently utilizing all CPUs and GPUs resources on the CAVE cluster • We proposed an accurate algorithm for the LCD panel simulation and rendering based on measured BRDF and emission data.

30. Future work • Use HDR video stream as dynamic environment map lighting • Trivial to do, we are just waiting for a fish-eye lens suitable for our HDR camera • Use HDR display (0.05-3,000 cd/m2) to render the LCD panel with luminance levels similar to real world driving conditions • Display-in-display rendering problem • Consider recently proposed techniques for all frequency PRT lighting

31. Acknowledgements • We would like to thank the following people • Michael Arnold (Virtual Reality Center, DaimlerChrysler AG) • Thomas Ganz (Research & Technology Displays and Controls (RBP/BM), DaimlerChrysler AG) • Matthias Bues (VR Lab., Fraunhofer IAO)

32. Adaptive Logarithmic Tone Mapping

33. Tone Mapping: Time-Dependent Visual Adaptation • Light adaptation • Dark adaptation