Recovering Geometric, Photometric and Kinematic Properties from Images

1. Recovering Geometric, Photometric and Kinematic Properties from Images Jitendra Malik Computer Science Division University of California at Berkeley Work supported by ONR, Interval Research, Rockwell, MICRO, NSF, JSEP

2. Physics of Image Formation • Lighting • BRDFs • Shape and Spatial layout • Internal DOFs Images

3. Solving inverse problems requires models • Define suitable parametric models for geometry, lighting, BRDFs, and kinematics. • Recover parameters using optimization techniques. • Humans better at selecting models; computers at recovering parameters.

4. But there will always be unmodeled detail….. • Models are always approximate. • Adding more parameters doesn’t help; data will be insufficient to recover these parameters.

5. Hybrid Approaches are best! • ANALYSIS • use images to recover a subset of object parameters. These are chosen judiciously so that they can be recovered robustly • SYNTHESIS • render using appropriately selected images or subimages, transformed using the model.

6. Talk Outline • Geometry • Debevec, Taylor and Malik, SIGGRAPH 96 • Photometry • Yu and Malik, SIGGRAPH 98 • Debevec and Malik, SIGGRAPH 97 • Kinematics • Bregler and Malik, CVPR 98

7. Modeling and Rendering Architecture from Photographs George Borshukov Yizhou Yu Paul Debevec Camillo Taylor Jitendra Malik Computer Vision Group Computer Science Division University of California at Berkeley

8. Overview • Photogrammetric Modeling • Allows the user to construct a parametric model of the scene directly from photographs • Model-Based Stereo • Recovers additional geometric detail through stereo correspondence • View-Dependent Texture-Mapping • Renders each polygon of the recovered model using a linear combination of three nearest views

9. Our Modeling Method: • The userrepresents the scene as a collection of blocks • The computersolves for the sizes and positions of the blocks according to user-supplied edge correspondences

10. Block Model User-Marked Edges Recovered Model

11. Arc de Triomphe Modeled from five photographs by George Borshukov

12. Surfaces of Revolution Taj Mahal modeled from one photograph by G. Borshukov

13. Synthetic View Photograph Recovered Model

14. Recovering Additional Detailwith Model-Based Stereo • Scenes will have geometric detail not captured in the model • This detail can be recovered automatically through model-based stereo

15. Scene with Geometric Detail Approximate Block Model

16. Model-Based Stereo • Given a key and an offset image, • Project the offset image onto the model • View the model through the key cameraWarped offset image • Stereo becomes feasible between key and warped offset images because: • Disparities are small • Foreshortening is greatly reduced

17. Key Image Warped Offset Image Offset Image Disparity Map

18. Synthetic Views ofRefined Model Four images composited with View-Dependent Texture Mapping

19. Rendering with View-DependentTexture Mapping • Triangulate the view hemisphere • For each polygon, determine which images viewed it from which angles • Label each triangle vertex according to best viewed image 2 5 1 4 3 view hemisphere

20. Rendering with View-DependentTexture Mapping • To render, determine to which triangle the viewpoint belongs • Compute Barycentric weights for the triangle vertices • Render the polygon with a weighted average of the three vertex images 2 5 1 4 3 view hemisphere

21. The Campanile (Debevec et al) • 20 photographs used • approx. 1-2 weeks of modeling time. • Real time rendering

24. Recovered Campus Model Campanile + 40 Buildings