Real-time Acquisition and Rendering of Large 3D Models

1. Real-time Acquisition and Rendering of Large 3D Models Szymon Rusinkiewicz

2. Shape Rendering 3D Scanning Computer Graphics Pipeline Shape • Human time = expensive • Sensors = cheap • Computer graphics increasingly relies onmeasurements of the real world Motion Lighting and Reflectance

3. Computer graphics Product inspection Robot navigation As-built floorplans Product design Archaeology Clothes fitting Art history 3D Scanning Applications

4. The Digital Michelangelo Project • Push state of the art in range scanning and demonstrate applications in art and art history Working in the museum Scanning geometry Scanning color

5. Traditional Range Scanning Pipeline • High-quality, robust pipeline for producing 3D models: • Scan object with laser triangulation scanner: many views from different angles • Align pieces into single coordinate frame:initial manual alignment, refined with ICP • Merge overlapping regions: compute “average” surface using VRIP [Curless & Levoy 96] • Display resulting model

6. 3D Scan of David: Statistics • Over 5 meters tall • 1/4 mm resolution • 22 people • 30 nights of scanning • Efficiency max : min = 8 : 1 • Needed view planning • Weight of gantry: 800 kg • Putting model together:1000+ man-hours and counting

7. Real-Time Model Acquisition New 3D Scanning Pipeline • Need for a fast, inexpensive,easy-to-use 3D scanning system • Wave a (small, rigid) object by hand in front of the scanner • Automatically align data asit is acquired • Let user see partial model as itis being built – fill holes

8. Real-Time 3D Model Acquisition • Prototype real-time model acquisition system • 3D scanning of moving objects • Fast alignment • Real-time merging and display

9. Applications of Easy-to-Use3D Model Acquisition • Advertising • More capabilities in Photoshop • Movie sets • Augmented reality • User interfaces

10. 3D Scanning Technologies • Contact-based: touch probes • Passive: shape from stereo, motion, shading • Active: time-of-flight, defocus, photometric stereo, triangulation • Triangulation systems are inexpensive, robust, and flexible • Take advantage of trends in DLP projectors

11. Laser Camera Laser Triangulation • Project laser stripe onto object Object

12. Laser Camera Laser Triangulation • Depth from ray-plane triangulation Object (x,y)

13. Triangulation • Faster acquisition: project multiple stripes • Correspondence problem: which stripeis which?

14. Multi-stripe Multi-frame Single-stripe Single-frame Triangulation Slow, robust Fast, fragile

15. Time-Coded Light Patterns • Assign each stripe a unique illumination codeover time [Posdamer 82] Time Space

16. Gray-Code Patterns • To minimize effects of quantization error:each point may be a boundary only once Time Space

17. Structured-Light Assumptions • Structured-light systems make certain assumptions about the scene: • Spatial continuity assumption: • Assume scene is one object • Project a grid, pattern of dots, etc. • Temporal continuity assumption: • Assume scene is static • Assign stripes a code over time

18. Codes for Moving Scenes • We make a different assumption: • Object may move • Velocity low enough to permit tracking • “Spatio-temporal” continuity

19. Illumination history = (WB),(BW),(WB) Code Codes for Moving Scenes • Code stripe boundariesinstead of stripes • Perform frame-to-frametracking of correspondingboundaries • Propagate illumination history [Hall-Holt & Rusinkiewicz, ICCV 2001]

20. New Scanning Pipeline Project Code Capture Images Find Boundaries Match Boundaries Decode Compute Range

21. Designing a Code • Biggest problem is ghosts – WW or BB “boundaries” that can’t be seen directly Project Code Capture Images Find Boundaries Match Boundaries Decode Compute Range

22. Designing a Code • Design a code to make tracking possible: • Do not allow two spatially adjacent ghosts • Do not allow two temporally adjacent ghosts t

23. 0000 1101 • Nodes: stripes (over time) 1010 0111 • Edges: boundaries (over time) 1111 0010 0101 1000 1011 0110 0001 1100 0100 1001 Time 1110 0011 Space Designing a Code • Graph (for 4 frames):

24. Path with alternating colors:55 edges in graph maximal-length traversal has 110 boundaries (111 stripes) 0000 1101 • Nodes: stripes (over time) 1010 0111 1111 0010 0101 1000 Boundary visible at even times Boundary visible at odd times 1011 0110 0001 1100 0100 1001 1110 0011 Designing a Code • Graph (for 4 frames): • Edges: boundaries (over time)

25. Image Capture • Standard video camera: fields at 60 Hz • Genlock camera to projector Project Code Capture Images Find Boundaries Match Boundaries Decode Compute Range

26. Finding Boundaries Project Code Capture Images Find Boundaries Match Boundaries Decode Compute Range • Standard edge detection problem • Current solution: find minima and maxima of intensity, boundary is between them

27. Matching Stripe Boundaries Project Code Capture Images Find Boundaries Match Boundaries Decode Compute Range • Even if number of ghosts isminimized, matching is not easy ?

28. Matching Stripe Boundaries • Resolve ambiguity by constraining maximum stripe velocity • Could accommodate higher speeds by estimating velocities • Could take advantage of methods intracking literature (e.g., Kalman filters)

29. Decoding Boundaries Project Code Capture Images Find Boundaries Match Boundaries Decode Compute Range • Propagate illumination history • Table lookup based on illumination history and position in four-frame sequence • Once a stripe has been tracked for at least four frames,it contributes useful data on every subsequent frame

30. Computing 3D Position • Ray-plane intersection • Requires calibration of: • Camera, projector intrinsics • Relative position and orientation Project Code Capture Images Find Boundaries Match Boundaries Decode Compute Range

31. Results Video frames Stripe boundaries unknown known ghosts

32. Results • Single range image of moving object Top View Front View Top View Front View Boundary codes and tracking Gray codes, no tracking

33. Aligning 3D Data • This range scanner can be used for any moving objects • For rigid objects, range images can be aligned to each other as object moves

34. Aligning 3D Data • If correct correspondences are known,it is possible to find correct relative rotation/translation

35. Aligning 3D Data • How to find corresponding points? • Previous systems based on user input,feature matching, surface signatures, etc.

36. Aligning 3D Data • Alternative: assume closest points correspond to each other, compute the best transform…

37. Aligning 3D Data • … and iterate to find alignment • Iterated Closest Points (ICP) [Besl & McKay 92] • Converges if starting position “close enough“

38. ICP Variants • Classic ICP algorithm not real-time • To improve speed: examine stages of ICP and evaluate proposed variants [Rusinkiewicz & Levoy, 3DIM 2001] • Selecting source points (from one or both meshes) • Matching to points in the other mesh • Weighting the correspondences • Rejecting certain (outlier) point pairs • Assigning an error metric to the current transform • Minimizing the error metric

39. ICP Variant – Point-to-Plane Error Metric • Using point-to-plane distance instead of point-to-point lets flat regions slide along each other more easily [Chen & Medioni 91]

40. Finding Corresponding Points • Finding closest point is most expensive stage of ICP • Brute force search – O(n) • Spatial data structure (e.g., k-d tree) – O(log n) • Voxel grid – O(1), but large constant, slow preprocessing

41. Finding Corresponding Points • For range images, simply project point [Blais 95] • Constant-time, fast • Does not require precomputing a spatial data structure

42. High-Speed ICP Algorithm • ICP algorithm with projection-based correspondences, point-to-plane matchingcan align meshes in a few tens of ms.(cf. over 1 sec. with closest-point)

43. Anchor Scans • Alignment of consecutive scans leads to accumulation of ICP errors • Alternative: align all scans to an “anchor” scan, only switch anchor when overlap low • Given anchor scans, restart after failed ICP becomes easier

44. Merging and Rendering • Goal: visualize the model well enoughto be able to see holes • Cannot display all the scanned data – accumulates linearly with time • Standard high-quality merging methods:processing time ~ 1 minute per scan

45. Merging and Rendering • Real-time incremental merging and rendering: • Quantize samples to a 3D grid • Maintain average normal of all pointsat a grid cell • Point (splat) rendering • Can be made hierarchical to conserve memory

46. Photograph

47. Real-time Scanning Demo

48. Postprocessing • Goal of real-time display is to let user evaluate coverage, fill holes • Quality/speed tradeoff • Offline postprocessing for high-quality models

49. Merged Result Photograph Aligned scans Merged

50. Future Work • Technological improvements: • Use full resolution of projector • Higher-resolution cameras • Ideas from design of single-stripe 3D scanners • Pipeline improvements: • Better detection of failed alignment • Better handling of object texture – combine with stereo? • Global registration to eliminate drift • More sophisticated merging • Improve user interaction during scanning