Hardware-Accelerated Silhouette Matching

1. Hardware-AcceleratedSilhouette Matching Hendrik Lensch, Wolfgang Heidrich, and Hans-Peter Seidel Max-Planck-Institut für Informatik, Saarbrücken (Germany)

2. Overview • Motivation • Comparing Silhouettes • Stitching and Combining Textures • Results and Conclusions

3. Geometry 3D scanner Texture data digital camera Acquiring Real World Models single sensor vs. multiple sensors

4. 3D – 2D Registration • Find the camera setting for each 2D image.

5. Camera Model • Transformations • to camera coordinates (extrinsic): • to 2D image space (intrinsic):  determine R, t and f (6+1 dimensions)

6. Similarity Measure • Which features to investigate? • no color information on the model • correspondence of geometric features hard to find

7. Similarity Measure • Compare silhouettes [Etienne de Silhouette 1709-1767] • model: render monochrome • photo: automatic histogram-based segmentation

8. Similarity Measure • Compare silhouettes [Etienne de Silhouette 1709-1767] • model: render monochrome • photo: automatic histogram-based segmentation

9. Point-to-outline distances slow because points on the outline must be determined speedup by distance maps Distance Measure for Silhouettes

10. Count the number of pixels covered by just one silhouette. XOR the images compute histogram (hardware) gives linear response to the displacement intensity x 0 difference 1 x 0 displacement Pixel-based Distance Measure 1

11. Count the number of pixels covered by just one silhouette. XOR the images compute histogram (hardware) gives linear response to the displacement intensity 1 1 x 0 difference 1 x 0 Pixel-based Distance Measure displacement

12. Use smooth transitions blur images integrate squared differences faster convergence reduced variance higher evaluation cost Approximation ofSquared Distances

13. Use smooth transitions blur images integrate squared differences faster convergence reduced variance higher evaluation cost filtersize intensity intensity 1 1 1 x x 0 0 difference 1 1 x 0 0 Approximation ofSquared Distances 1 difference x

14. Non-linear Optimization • Downhill Simplex Method [Press 1992] • works for N dimensions • no derivatives • easy to control

15. Simplex Method in 3D original simplex reflection and/or expansion shrinking random perturbation

16. Hierarchical Optimization • optimize on low resolution first • restart optimization to avoid local minima • switch to higher resolution • mesh resolution can be adapted

17. Starting Point Generation • set camera distance tz depending on object size • settx and ty to zero • select 48 sample rotations • run optimization for each of the samples (40 evaluations) • select top 5 results • restart optimization (200 evaluations) • take best result as starting point

18. Texture Stitching • projective texture mapping • assign one image to each triangle • triangle visible in image? (test every vertex) • select best viewing angle • discard data near depth discontinuities

19. find border vertices release all triangles around them assign boundary vertices to best region assign alpha-values for each region 1 to vertices included in the region 0 to all others. Blending Across Assignment Borders

20. Entire Texture

21. Results and Conclusions • Problems solved: • automatic texture registration (R, t, f) • view-independent texture stitching • blending across assignment boundaries • rough manual alignment helps (speedup, failures) • Further problems: • extract purely diffuse part of texture • generate texture where data is missing

22. Questions? • visit us at • www.mpi-sb.mpg.de