Rendering Synthetic Objects into Real Scenes:

1. Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography Paul Debevec, University of California at Berkley

2. Motivation • Visual Effects • Adding synthetic actors/props • Architectural simulation • Adding buildings to landscapes

3. Related Work • Previous methods used to insert synthetic objects into real scenes • Drawing • Environment mapping • Approximate geometric model and lighting • Global illumination algorithms • Recovery of HDR radiance maps from photographs

4. Related Work • High Dynamic Range refers to the range of brightness values present in a scene • Eyes, photos, displays, all capture a limited, non-linear range 1000:1 100:1 Everyday brightness range (about 1012:1) Source: Microsoft Developers Network

5. Related Work • Why is this a problem? • Limited brightness range results in lost scene detail from saturation and/or underexposure • OK for traditional image use Photos from OpenEXR by LucasFilm, LTD. http://www.openexr.com/samples.html

6. Related Work • What about image-based lighting? • Photo pixel values are not representative of relative Radiance in the scene Images by Paul Debevec, 98

7. Related Work • Solution: Take multiple photos with varying shutter speeds and ‘combine’ somehow to create accurate relative radiance maps Images by Paul Debevec, 98

8. Related Work • Is HDR really important?

9. Method • Divide the scene into three parts

10. Method • Distant Scene • Light-based model • Required to provide correct incident illumination to local/synthetic objects from desired viewpoint • Assumed that no light from the model will affect the distant scene • Any level of geometric detail wanted

11. Method • Local Scene • Material-based model (has BRDF) • Must approximate the real scene local to the position of the synthetic objects • ‘Local’ means photometrically local • ‘Approximate’ means geometrically AND photometrically representative of the real scene

12. Method • Synthetic Objects • Complete materials-based models of elements that are to be inserted into the scene • Geometric and lighting information • Any shape and material supported by the global illumination method that you plan to use

13. Method • Compositing using a light probe • One method to create the distant scene from a real location • Uses a metal sphere and a camera to gather light from a point

14. Method • Compositing using a light probe • This light is mapped onto the distant scene geometry and used to calculate the lighting • Lighting is calculated with global illumination

15. Method • Results

16. Method • Problem with the current definition of the local scene • We said that it must approximate the material of the real scene • It can be really hard to get the BRDF/material of a real object! • Can we modify the local scene so it doesn’t require this?

17. Method • Local Scene 2 (Differential Rendering) • Approximate the material of the scene to within reason (not exact) • Render with and without your synthetic objects and calculate the difference • Add that difference back into the original image

18. Method • Difference between local scene render with and without synthetic objects - =

19. Results

20. Conclusion • This paper presented a framework for rendering synthetic objects into real world environments with real lighting • High Dynamic Range radiance maps • Global Illumination • Distant and local scenes, synthetic objects • Differential rendering

21. Future Work • Method to generate boundary between distant and local scene • More efficient global illumination

22. Comments • Pros • Open framework • Good results • Cons • Hard to decide what local scene should be! • Local scene can be very complicated