Lightfields, Lumigraphs, and Image-based Rendering

1. Lightfields, Lumigraphs, and Image-based Rendering

2. 2D to 4D • More than can be captured in one image • All light passing through a region of space • General (5D) vs. unoccluded space (4D) “Rebinning” pixels

3. Image-Based Modeling and Rendering • Generate new views of a scene directly from existing views • “Pure” IBR (such as lightfields): no model of scene • Other IBR techniques try to obtain higher quality with less storage by building a model

4. Lightfields • Advantages: • Simpler computation vs. traditional CG • Cost independent of scene complexity • Cost independent of material properties and other optical effects • Disadvantages: • Static geometry • Fixed lighting • High storage cost

5. Using Lightfields • Obtain 2D slices of 4D data set • Arbitrary views: take other 2D slices • Challenges: • Capture • Parameterization • Compression • Rendering

6. Capturing Lightfields • Need a 2D set of (2D) images • Choices: • Camera motion: human vs. computer • Constraints on camera motion • Coverage and sampling uniformity • Aliasing

7. Lightfield Parameterization • Point / angle • Two points on a sphere • Points on two planes • Original images and camera positions

8. Compression • Compress individual images (JPEG, etc.) • Adapt video compression to 2D arrays • Decomposition into basis functions • Vector quantization

9. Rendering • How to select rays? • Interpolation • Taking advantage of hardware • Graphics hardware • Compression hardware

10. Implementations • Lightfields, SIGGRAPH 96,Levoy and Hanrahan • Lumigraphs, SIGGRAPH 96,Gortler et al. • Unstructured lumigraphs, SIGGRAPH 01, Buehler et al.

11. Light Field Rendering • Capture: • Computer-controlled camera rig • Move camera to grid of locations on a plane

12. Light Field Rendering • Parameterization: • Two planes, evenly sampled: “light slab” • In general, planes in arbitrary orientations • In practice, one plane = camera locations • Minimizes resampling

13. Light Field Coverage

14. Multi-Slab Light Fields

15. Rendering • For each desired ray: • Compute intersection with (u,v) and(s,t) planes • Take closest ray • Variants: interpolation • Bilinear in (u,v) only • Bilinear in (s,t) only • Quadrilinear in (u,v,s,t)

16. Light Field Compression • Based on vector quantization • Preprocessing: build a representative codebook of 4D tiles • Each tile in lightfield represented by index • Example: 2x2x2x2 tiles, 16 bit index = 24:1 compression

17. Lightfield Compression byVector Quantization • Advantages: • Fast at runtime • Reasonable compression • Disadvantages: • Preprocessing slow • Manually-selected codebook size • Does not take advantage of structure

18. The Lumigraph • Capture: move camera by hand • Camera intrinsics assumed calibrated • Camera pose recovered from markers

19. Lumigraph Postprocessing • Obtain rough geometric model • Chroma keying (blue screen) to extract silhouettes • Octree-based space carving • Resample images to two-plane parameterization

20. Lumigraph Rendering • Use rough depth information to improve rendering quality

21. Lumigraph Rendering • Use rough depth information to improve rendering quality

22. Without usinggeometry Using approximate geometry Lumigraph Rendering

23. Unstructured Lumigraph Rendering • Further enhancement of lumigraphs:do not use two-plane parameterization • Store original pictures: no resampling • Hand-held camera, moved around an environment

24. Unstructured Lumigraph Rendering • To reconstruct views, assign penalty to each original ray • Distance to desired ray, usingapproximate geometry • Resolution • Feather near edges of image • Construct “camera blending field” • Render using texture mapping

25. Unstructured Lumigraph Rendering Blending field Rendering

26. Other Lightfield Acquisition Devices • Spherical motionof camera aroundan object • Samples space ofdirections uniformly • Second arm tomove light source –measure BRDF

27. Other Lightfield Acquisition Devices • Acquire an entirelight field at once • Video rates • Integrated MPEG2compression foreach camera (Bennett Wilburn, Michal Smulski, Mark Horowitz)