Advanced 3D Model Construction for Visualization at UC Berkeley's AVID Lab

1. Three Dimensional Model Construction for Visualization Avideh Zakhor Video and Image Processing Lab University of California at Berkeley avz@eecs.berkeley.edu

2. Outline • Goals and objectives • Previous work by PI • Directions for future work

3. Goals and Objectives • Develop a framework for fast, automatic and accurate 3D model construction for objects, scenes, rooms, buildings (interior and exterior), urban areas, and cities. • Models must be easy to compute, compact to represent and suitable for high quality view synthesis and visualization • Applications: Virtual or augmented reality fly-throughs.

4. Previous Work on Scene Modeling • Full/Assisted 3-D ModelingKanade et al.; Koch et al.; Becker & Bove; Debevec et al.; Faugeras et al.; Malik & Yu. • Mosaics and PanoramasSzeliski & Kang; McMillan & Bishop; Shum & Szeliski • Layered/LDI RepresentationsWang & Adelson; Sawhney & Ayer; Weiss; Baker et al. • View Interpolation/IBR/Light FieldsChen & Williams; Chang & Zakhor; Laveau & Faugeras; Seitz & Dyer; Levoy & Hanrahan

5. Previous Work on Building Models • Nevatia (USC): multi-sensor integration • Teller (MIT): spherical mosaics on a wheelchair sized rover, known 6DOF • Van Gool (Belgium): roof detection from aerial photographs • Peter Allen (Columbia): images and laser range finders; view/sensor planning. • Faugeras (INRIA)

6. Previous Work on City Modeling • Planet 9: • Combines ground photographs with existing city maps manually. • UCLA Urban Simulation Team: • Uses mutligen to create models from aerial photographs, together with ground video for texture mapping. • Bath and London models by Univ. of Bath. • Combines aerial photgraphs with existing maps. • All approaches are slow and labor intensive.

7. Work at VIP lab at UCB Scene modeling and reconstruction.

8. Multi-Valued Representation: MVR • Level k has k occluding surfaces • Form multivalued array of depth and intensity

9. Observations

10. Imaging geometry (1) • Planar translation

11. Imaging Geometry (2) • Circular/orbital motion

12. Dense Depth Estimation • Estimate camera motion • Compute depth maps to build MVRs • Low-contrast regions problematic for dense depth estimation. • Enforce spatial coherence to achieve realistic, high quality visualization.

13. Block Diagram for Dense Depth Estimation • Planar approximation of depth for low contrast regions.

14. Oroginal Sequences “Mug” sequence (13 frames) “Teabox” sequence (102 frames)

15. Low-Contrast Regions • Complete tracking Mug sequence Tea-box sequence

16. Multiframe Depth Estimation Apply iterative estimation algorithm to enforce piecewise smoothness, without smoothing over depth discontinuities.

17. Multiframe Depth Estimation Mug Tea-box Multiframe Stereo + Low-Contrast Processing + Piecewise Smoothing Multiframe Stereo + Low-Contrast Processing + Piecewise Smoothing

18. Multivalued Representation • Project depths to reference coordinates

19. Multivalued representation for frame 4 (Level 0) Results (1) • Mug sequence

20. Multivalued representation for frame 4 (Level 1) Results • Mug sequence

21. Multivalued representation for frame 4 (Combining Levels 0 and 1) Results • Mug sequence

22. Results • Mug sequence Reconstructed sequence Arbitrary flythrough

23. Results (2) • Teabox sequence Multivalued representation for frame 22 (Intensity, Level 0)

24. Results • Teabox sequence Multivalued representation for frame 22 (Depth, Level 0)

25. Results • Teabox sequence Multivalued representation for frame 22 (Intensity, Level 1)

26. Results • Teabox sequence Multivalued representation for frame 22 (Depth, Level 1)

27. Results • Teabox sequence Multivalued representation for frame 22 (Intensity, combining Levels 0 and 1)

28. Results • Teabox sequence Multivalued representation for frame 22 (Depth, combining Levels 0 and 1)

29. Results • Teabox sequence Multivalued representation for frame 86 (Intensity, Level 0)

30. Results • Teabox sequence Multivalued representation for frame 86 (Depth, Level 0)

31. Results • Teabox sequence Multivalued representation for frame 86 (Intensity, Level 1)

32. Results • Teabox sequence Multivalued representation for frame 86 (Depth, Level 1)

33. Results • Teabox sequence Multivalued representation for frame 86 (Intensity, combining Levels 0 and 1)

34. Results • Teabox sequence Multivalued representation for frame 86 (Depth, combining Levels 0 and 1)

35. Multiple MVRs • Perform view interpolation w/many MVRs

36. Results: multiple MVRs • Teabox sequence Reconstructed sequence from MVR86 Reconstruct sequence from MVR22

37. Results: Multiple MVRs Reconstructed sequence Arbitrary flyaround

38. Extensions • Complex scenes with many “levels” are difficult to model with MVR; e.g. trees, leaves, etc • Difficult to ensure realistic visualization from all angles; Need to plan capture process carefully. • Tradeoff between CG polygon modeling and IBR; • Use both in real visualization databases. • Build polygon models from MVR.

39. Issues for model construction • Choice of geometry for obtaining data • Choice of imaging technology. • Choice of representation. • Choice of models. • Dealing with time varying scenes.

40. Extensions: • So far, addressed “outside in” problem: • Camera looked inward to “scan” the object. • Future work will focus on the “Inside out” problem: • Modeling a room, office. • Modeling exterior or interior of a building • Modeling an urban environment e.g. a city

41. Strategy • Use: • Range sensors, position sensors (GPS), Gyros(orientation), omni camera, video. • Existing datasets: 3D CAD models, digital elevation maps (DEM), DTED, city maps, architectural drawings: apriori information

42. Modeling interior of buildings • Leverage existing work in the computer graphics group at UCB: • 3D model of Soda hall available from the “soda walkthrough” project. • 3D model built out of architectural drawings • Use additional video, and laser range finder input to • Enhance the details of the 3D model: furniture, etc • Add texture maps for photo-realistic walk-throughs.

43. City Modeling • Develop a framework for modeling parts of city of San Francisco: • Use aerial photograph as provided by Space Imaging Corp; resolution 1 ft. • Use digitized city maps • Use ground data collection vehicle to collect range and intensity video from a panoramic camera, annotated with 6 DOF parameters. • Derive data fusion algorithms to process the above in speedy, automated and accurate fashion.

44. Requirements • Automation (little or no interaction needed from human operators) • Speed: must scale with large areas and large data sets. • Accuracy • Robustness to location of data collection. • Ease of data collection. • Representation suitable to hierarchical visualization databases.

45. Relationship to others • USC: accurate tracking and registration algorithms needed for model construction. • Syracuse: uncertainty processing, and data fusion for model construction. • G. Tech: How to combine CG polygonal model building with IBR models in vis. database? How can vis. databases deal with photo-realistic rendering?

46. Conclusions • Fast, accurate and automatic model construction is essential to mobile augmented reality systems. • Our goal is to provide photo-realistic rendering of objects, scenes, buildings, and cities, to enable, visualization, navigation and interaction.