Reconstructing Building Interiors from Images

1. Reconstructing Building Interiors from Images Yasutaka Furukawa Brian Curless Steven M. Seitz University of Washington, Seattle, USA 2011/01/16 蔡禹婷

2. Outline • Introduction • Goal • Challenges • System pipeline • Algorithmic details (technical contribution) • Experimental Results • Conclusion and future work • Reference

3. Outline • Introduction • Goal • Challenges • System pipeline • Algorithmic details (technical contribution) • Experimental Results • Conclusion and future work • Reference

4. Reconstruction and Visualization of Architectural Scenes • Semi-automatic(Manual ) • Google Earth & Virtual Earth • Façade :  Building facade made for use as a real-time video game engine environment. • Automatic • Ground-level images • Aerial images: A projected image which is "floating in air", and cannot be viewed normally. Aerial images Google Earth 4  Virtual Earth

5. Reconstruction and Visualization of Architectural Scenes • Difficulty • Little attention paid to indoor scenes • If you walk inside your home and take photographs, generating a compelling 3D reconstruction and visualization becomes much more difficult. ? ? Google Earth 4  Virtual Earth ? ? Aerial images

6. Outline • Introduction • Goal • Challenges • System pipeline • Algorithmic details (technical contribution) • Experimental Results • Conclusion and future work • Reference

7. Goal • Fully automatic system for interiors / outdoors • Reconstructs a simple 3D model from images • Provides real-time interactive visualization

8. Outline • Introduction • Goal • Challenges • System pipeline • Algorithmic details (technical contribution) • Experimental Results • Conclusion and future work • Reference

9. Challenges • Reconstruction • Multi-view stereo (MVS) typically produces a dense model • We want the model to be • Simple for real-time interactive visualization of a large scene (e.g., a whole house) • Accurate for high-quality image-based rendering • Simple mode is effective for compelling visualization

10. Challenges • Indoor Reconstruction Texture-poor surfaces Complicated visibility Prevalence of thin structures (doors, walls, tables)

11. Outline • Introduction • Goal • Challenges • System pipeline • Algorithmic details (technical contribution) • Experimental Results • Conclusion and future work • Reference

12. System pipeline • 3D reconstruction and visualization system for architectural scenes.

13. System pipeline • Image-based • SFM • MVS • MWS • Merging

14. System pipeline Image-based Image-based SFM MVS MWS Merging Image-based

15. System pipeline Image-based SFM MVS MWS Merging Structure-from-Motion Bundler by Noah Snavely Structure from Motion for unordered image collections WEB: http://phototour.cs.washington.edu/bundler/

16. System pipeline Image-based SFM MVS MWS Merging Multi-view Stereo PMVS by Yasutaka Furukawa and Jean Ponce Patch-based Multi-View Stereo Software/

17. System pipeline Image-based SFM MVS MWS Merging Manhattan-world Stereo

18. System pipeline Image-based SFM MVS Manhattan-world Stereo

19. System pipeline Image-based SFM MVS MWS Merging Manhattan-world Stereo Result

20. System pipeline Image-based SFM MVS MWS Merging Axis-aligned depth map merging (Paper contribution)

21. Outline • Introduction • Goal • Challenges • System pipeline • Algorithmic details (technical contribution) • Experimental Results • Conclusion and future work • Reference

22. Axis-aligned Depth-map Merging • Basic framework is similar to volumetric MRF

23. Axis-aligned Depth-map Merging • Basic framework is similar to volumetric MRF

24. Key Feature 1 - Penalty terms

25. Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data

26. Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & dataUnary is often constant (inflation)

27. Key Feature 1 - Penalty terms • Weak regularization at interesting places • Focus on a dense model Binary penalty Binary encodes smoothness & dataUnary is often constant (inflation)

28. Key Feature 1 - Penalty terms • Weak regularization at interesting places • Focus on a dense model • We want a simple model Binary penalty Binary encodes smoothness & dataUnary is often constant (inflation)

29. Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)

30. Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation)

31. Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation) Unary encodes data

32. Key Feature 1 - Penalty terms Binary penalty Binary encodes smoothness & data Unary is often constant (inflation) Binary is smoothness Unary encodes data

33. Key Feature 1 - Penalty terms Binary penalty Regularization becomes weakDense 3D model Regularization is data-independent Simpler 3D model

34. Axis-aligned Depth-map Merging • Align-voxel grid withthe dominant axes

35. Axis-aligned Depth-map Merging • Align-voxel grid withthe dominant axes • Data term (unary)

36. Axis-aligned Depth-map Merging • Align voxel grid withthe dominant axes • Data term (unary)

37. Axis-aligned Depth-map Merging • Align voxel grid withthe dominant axes • Data term (unary)

38. Axis-aligned Depth-map Merging • Align voxel grid withthe dominant axes • Data term (unary) • Smoothness (binary)

39. Axis-aligned Depth-map Merging • Align voxel grid withthe dominant axes • Data term (unary) • Smoothness (binary)

40. Axis-aligned Depth-map Merging • Align voxel grid withthe dominant axes • Data term (unary) • Smoothness (binary) • Graph-cuts

41. Key Feature 2 – Regularization • For large scenes, data info are not complete • Typical volumetric MRFs bias to general minimal surface • We bias to piece-wise planar axis-aligned for architectural scenes

42. Key Feature 2 – Regularization

43. Key Feature 2 – Regularization

44. Key Feature 2 – Regularization

45. Key Feature 2 – Regularization

46. Key Feature 2 – Regularization

47. Key Feature 2 – Regularization Same energy (ambiguous)

48. Key Feature 2 – Regularization Same energy (ambiguous) Data penalty: 0

49. Key Feature 2 – Regularization Same energy (ambiguous) Data penalty: 0 Smoothness penalty: Data penalty: 0 Smoothness penalty: 24 Data penalty: 0

50. Key Feature 2 – Regularization shrinkage