1. Image-Based Rendering • Geometry and light interaction may be difficult and expensive to model • Think of how hard radiosity is • Imagine the complexity of modeling the exact geometry of carpet (as just one example) • Image based rendering seeks to replace geometry and surface properties with images

2. Texture Mapping • Replace small scale geometry and surface changes with an image attached to the surface • Define a mapping from surface points into a texture image • When the properties of that point are required, look up the pixel in the texture image • A research field in itself • Various methods of sampling the texture • Various ways of combining the samples with direct lighting calculations • Ways of arranging and accessing textures in memory

3. Environment Mapping • Replace the world around an object with images • Generally used for fast reflection computations • Typically, define a cube around the object and associate images with the inside of each face • When seeking reflection color, shot ray from surface point onto cube and take corresponding image point • Images represent a view of the world from the center of the object • Assumes that the world wouldn’t look much different if viewed from points on the object’s surface

4. Plenoptic Function • Describes the intensity of light: • passing through a given point, x • in a given direction, (,) • with given wavelength,  • at a given time, t • Many image-based rendering approaches can be cast as sampling from and reconstructing the plenoptic function • Note, function is generally constant along segments of a line (assuming vacuum)

5. Image-Based Rendering • Aim: Generate an image from a desired view using existing images from other views • May or may not know the viewing parameters for the existing images • Existing images may be photographs or computer generated renderings

6. A Plethora of Approaches • Methods differ in many ways: • The range of new viewpoints allowed • The density of input images • The representation for samples (known images) • The amount of user help required • The amount of additional information required (such as intrinsic camera parameters) • The method for gathering the input images

7. Movie-Map Approaches • Film views from fixed locations, closely spaced, and store on video-disc (would be DVD now) • Allow the user to jump from location to location, and pan • Appropriate images are retrieved from disk and displayed • No reprojection – just uses nearest existing sample • Still used in video games today, but with computer generated movies

8. Quicktime VR (Chen, 1995) • Movie-maps in software • Construct panoramic images by stitching together a series of photographs • Semi automatic process, based on correlation • Scale/shift images so that they look most alike • Works best with >50% overlap • Finite set of panoramas – user jumps from one to the other

9. View Interpolation (Chen and Williams, 1993) • Input: A set of synthetic images with known depth and camera parameters (location, focal length, etc) • Computes optical flow maps related each pair of images • Optical flow map is the set of vectors describing where each point in the first image moves to in the second image • Morphs between images by moving points along flow vectors • Intermediate views are “real” views only in special cases

10. View Morphing (Seitz and Dyer, 1997) • Uses interpolation to generate new views such that the intermediate views represent real camera motion • Observation: Interpolation gives correct intermediate views if the initial and final images are parallel views

11. View Morphing Process • Basic algorithm: • Pre-warp input images to get them into parallel views • Interpolate for intermediate view • Post-warp to get final result • User specifies a camera path that rotates and translates initial camera onto final camera • Requires knowledge of projection matrices for the input images • Found with vision algorithms. User may supply correspondences • Intermediate motion can be specified by giving trajectories of four points

12. Plenoptic Modeling (McMillan and Bishop, 1995) • Input: Set of panoramic images gathered by panning a video camera about its vertical optical axis on a tripod • The algorithm: • Determines the properties of each camera and registers the images associated with each pan • Determines the relative locations of each camera • Determines the depth of each point seen by multiple cameras • Generates new views by reconstructing the plenoptic function from the available samples

13. Fitting Each Camera • Problem: • Given several images all taken with the same camera rotated about its optical center • Determine the camera parameters • Determine the angle between each image • Approach: • Multi-stage optimization • First stage estimates angle between images and focal length • Second stage refines and gets remaining parameters • When done, can map a pixel in one image to its correct position in any other images

14. Locating the Cameras • Given cylindrical projections (panoramic images) for two cameras • User identifies 100-500 tie-points (points seen in both images) • Each tie-point defines two rays – one from the center of each camera through the tie-point • These rays should intersect at the world location for the point • Minimization step finds the camera locations and some other parameters that minimize the perpendicular distance between all the rays

15. Determining Disparity • The minimization algorithm gives us the world location of the tie-points, but what about the rest of the points in the image? • Use standard computer vision techniques to find the remaining disparities • Disparity is the angular difference between the locations of a point in two images • Directly related to the depth of the point • Makes heavy use of the epipolar constraint

16. The Epipolar Constraint • The location of a point in one image constrains it to lie on a ray in space passing through the camera and image point • This ray projects to a curve in the second image • Line for planar projection • Sine curve for cylindrical projection • Since the point must lie on the ray in world space, it must lie on the curve in the second image • Reduces the search for correspondences to a linear one along the line

17. Reconstructing a New View • The disparity from a known cylinder pair can be transferred to a new cylinder, and then reprojected onto a plane (the image) • Can all be done in one step • Have to resolve occlusion problems • Two points from the reference image could map to the same point in the output image • Solution: Define a fill ordering that guarantees correct occlusion (an important contribution of this paper) • Also have to fill holes

18. Summary • Image-based rendering obviously relies heavily on computer vision techniques • Particularly: Depth from stereo • The problem is very hard with real images • These techniques are not perfect! • Sampling remains a problem • Images tend to appear blurry • Relatively little work on reconstruction algorithms