Real-Time 3D Model Acquisition

1. Real-Time 3D Model Acquisition Princeton University Stanford University Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy

2. 3D Scanning

3. Possible Research Goals • Low noise • Guaranteed high accuracy • High speed • Low cost • Automatic operation • No holes

4. 3D Model Acquisition Pipeline 3D Scanner

5. View Planning 3D Model Acquisition Pipeline 3D Scanner

6. View Planning Alignment 3D Model Acquisition Pipeline 3D Scanner

7. View Planning Alignment Merging 3D Model Acquisition Pipeline 3D Scanner

8. View Planning Alignment Done? Merging 3D Model Acquisition Pipeline 3D Scanner

9. View Planning Alignment Done? Merging Display 3D Model Acquisition Pipeline 3D Scanner

10. 3D Model Acquisition Difficulties • Much (often most) time spent on “last 20%” • Pipeline not optimized for hole-filling • Not sufficient just to speed up scanner –must design pipeline for fast feedback

11. Real-Time 3D Model Acquisition

12. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Human Done? Merging Display

13. Alignment Merging Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Challenge: Real Time Done? Display

14. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Part I: Structured-LightTriangulation Done? Merging Display

15. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Part II: Fast ICP Done? Merging Display

16. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Part III: Voxel Grid Done? Merging Display

17. Laser Camera Triangulation • Project laser stripe onto object Object

18. Camera Triangulation • Depth from ray-plane triangulation Object Laser (x,y)

19. Triangulation • Faster acquisition: project multiple stripes • Correspondence problem: which stripeis which?

20. Multi-stripe Multi-frame Single-stripe Single-frame Continuum of Triangulation Methods Slow, robust Fast, fragile

21. Time-Coded Light Patterns • Assign each stripe a unique illumination codeover time [Posdamer 82] Time Space

22. Illumination history = (WB),(BW),(WB) Code Codes for Moving Scenes • Assign time codesto stripe boundaries • Perform frame-to-frametracking of correspondingboundaries • Propagate illumination history [Hall-Holt & Rusinkiewicz, ICCV 2001]

23. Designing a Code • Want many “features” to track:lots of black/white edges at each frame • Try to minimize ghosts – WW or BB “boundaries” that can’t be seen directly

24. Designing a Code 0000 1101 1010 0111 1111 0010 0101 1000 1011 0110 0001 1100 0100 1001 1110 0011 [Hall-Holt & Rusinkiewicz, ICCV 2001]

25. Implementation • Pipeline: • DLP projector illuminates scene @ 60 Hz. • Synchronized NTSC camera captures video • Pipeline returns range images @ 60 Hz. Project Code Capture Images Find Boundaries Match Boundaries Decode Compute Range

26. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Part II: Fast ICP Done? Merging Display

27. Aligning 3D Data • ICP (Iterative Closest Points): for each point on one scan, minimize distance to closest point on other scan…

28. Aligning 3D Data • … and iterate to find alignment • Iterated Closest Points (ICP) [Besl & McKay 92]

29. ICP in the Real-Time Pipeline • Potential problem with ICP: local minima • In this pipeline, scans close together • Very likely to converge to correct (global) minimum • Basic ICP algorithm too slow (~ seconds) • Point-to-plane minimization • Projection-based matching • With these tweaks, running time ~ milliseconds[Rusinkiewicz & Levoy, 3DIM 2001]

30. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Part III: Voxel Grid Done? Merging Display

31. Merging and Rendering • Goal: visualize the model well enoughto be able to see holes • Cannot display all the scanned data – accumulates linearly with time • Standard high-quality merging methods:processing time ~ 1 minute per scan

32. Merging and Rendering

33. Merging and Rendering

34. Merging and Rendering

35. + Merging and Rendering

36. Merging and Rendering • Point rendering, using accumulated normals for lighting

37. Example: Photograph 18 cm.

38. Result

39. Postprocessing • Real-time display • Quality/speed tradeoff • Goal: let user evaluate coverage, fill holes • Offline postprocessing for high-quality models • Global registration • High-quality merging (e.g., using VRIP [Curless 96])

40. Postprocessed Model

41. Recapturing Alignment

42. Summary • 3D model acquisition pipeline optimized for obtaining complete, hole-free models • Use human’s time most efficiently • Pieces of pipeline selected for real-time use: • Structured-light scanner for moving objects • Fast ICP variant • Simple grid-based merging, point rendering

43. Limitations • Prototype noisier than commercial systems • Could be made equivalent with careful engineering • Ultimate limitations on quality: focus, texture • Scan-to-scan ICP not perfect  alignment drift • Due to noise, miscalibration, degenerate geometry • Reduced, but not eliminated, by “anchor scans” • Possibly combine ICP with separate trackers

44. Future Work • Faster scanning • Better stripe boundary tracking • Multiple cameras, projectors • High-speed cameras, projectors • Application in different contexts • Cart- or shoulder-mounted for digitizing rooms • Infrared for imperceptibility

45. Acknowledgments • Collaborators: • Li-Wei He • James Davis • Lucas Pereira • Sean Anderson • Sponsors: • Sony • Intel • Interval