Image-Based Modeling, Rendering and Lighting

1. Image-Based Modeling, Rendering and Lighting Hubert Shum http://info.hubertshum.com

2. Overview Why we need image-based modeling and rendering Techniques in image-based modeling and rendering Mixed reality with image based lighting

3. Why We Need Image-Based Modeling and Rendering

4. Traditional Computer Graphics Image View Point Models

5. Traditional Computer Visions Model Real Objects Multiple RealPhotographs

6. Synthesizing Imaginary View From Photos -By Computer Graphics and Visions Real Objects Image In New View Model Multiple RealPhotographs View Point

7. Problem 1 – Visions Generate Noisy Model Real Objects NoisyModel Image Multiple RealPhotographs View Point

8. Problem 2 –Graphics Create Unrealistic Images UnrealisticImage Model View Point

9. Synthesizing Imaginary View From Photos -Image Based Modeling and Rendering ? Real Objects Image In New View IntermediateRepresentations Multiple RealPhotographs

10. Advantages • Fast and realistic • “Photorealistic” lighting and shadows from images • No need to synthesize lighting and shadows • No need to create complex models • Able to change view points as if 3D

11. Techniques in Image Based Modeling and Rendering

12. Techniques In IBMR Geometry-based method Image-based method Light field rendering

13. Geometry-Based Method • Stereo vision based on points and lines produce noisy models • Take advantage on our objective:Synthesize an image in a new viewing angle • Make assumption to simplify the complexity • Polyhedral approach • Layers approach

14. Geometry-Based Method –Polyhedral Approach Reconstructing Polyhedral Models of Architectural Scenes from PhotographsTaylor et al. Model the object with polyhedral manually Apply model-based stereo vision Render image of new views based on model

15. Geometry-Based Method –Layer Approach A Layered Approach to Stereo ReconstructionBaker et al. Apply stereo vision to extract the depth of layers, rather than the depth of points Render new views by moving layers of objects

16. Image-Based Method • Do not construct geometric models • Find out correspondence between images • Interpolate images to portray 3D scenes • Image wrapping • View interpolation • View morphing

17. Image-Based Method - Image Wraping Capture multiple images from different angles Mosaicing images to form a continuous large image Map mosaiced image onto a cylinder / cube Example: QuickTime VR

18. Image-Based Method - Image Wraping Video

19. Image-Based Method - View Interpolation View Interpolation for Image SynthesisChen et al. • Capture two images by moving the camera • Correspond pixels in the two image • Since the two images are different only in camera movement, correspondence is uniform • Generate pixel offset vectors • Create new views by interpolating pixel offset vectors • May create gaps since some areas are not present in both images • Filling in gaps by interpolation

20. Image-Based Method - View Morphing View MorphingSeitz et al. Improve distortion in image warping method Assume two cameras is setup with pre-defined criteria Capture images from both cameras Estimate the projection relationship between the two cameras Warp the images based on the projection matrix to minimize distortion

21. Light Field Rendering • Consider the distribution of light in a space • E.g. A blue object reflect blue light to a point in the space • Model all the light in the space such that we know the light at any position in any angles

22. Light Field Rendering – Radiance Fundamentals of Image Formation and Re-use Sillor et al. • The radiance leaving a point x at angle Θ is defined as: • Light emitted by point x • Light reflected by point x from all other sources • Radiance is defined for each wavelength (color) of light • p is the bi-directional reflectance distribution function (BRDF) • Define the amount of light to be reflected

23. Light Field Rendering – Light Field • Light field describes the radiance at every point in the space • It is a complex function dependent on the position of the point and the viewing angle • Given the light field in a space • We know the light information of every point and every angle in the space • We can render the objects in the space in different view points • However, calculating the light field numerically is very difficult

24. Light Field Rendering – Model with Real Images Light Field RenderingLevoy et al. • Capture the image of an object for all quantized positions and viewing angles • Model the light field based on the resultant image sequences • The radiance of a point at the object becomes a function of intensity for all images • Synthesize the image from any view point

25. Light Field Rendering – Model with Real Images Video

26. Mixed Reality with Image Based Lighting

27. Real / Synthesized? Image Based LightingDebevec et al.

28. What It Does • Given a virtual space generated by image warping / other image based rendering technique • Objective: put a virtual object in the space • The virtual object should reflect the light from the virtual environment

29. How It Works – Capture Environment • Capture real-world illumination as images • Images has to be omni-directional • Pixel values are linearly proportional to the light in the real world • Two capture methods • Mirror ball • Mosaic images of different views

30. How It Works – Virtual Objects • Then, we create virtual objects • Same process as computer graphics • Setup the reflection parameters • If there are multiple objects, they may reflect light from each other

31. How It Works – Combining Environment and Virtual Objects Place the environment to surround the virtual objects The intensity of the environment is represented by the images Apply global illumination to determine the light of each point at the virtual objects

32. Final Results

33. Image Based Lighting Demo

34. Conclusion • Image based techniques • Different from computer graphics and visions • Advantages • Image based modeling and rendering • Different methods • Image based lighting • Putting virtual objects to scene created from real images