3D Shape Retrieval from Silhouette Images

1. 3D Shape Retrieval from Silhouette Images Nat Duca

2. Overview (1) Input: Output: Edges:

3. Overview (2) • Are silhouettes good descriptions of shape? • One may not be… but a few are • Can you see 3D structure in the shadow of a hand in the projector? • Silhouettes are easily entered queries… draw the contour of your shape • Also, they can be derived from an arbitrary sketch or image… or shadows, come to think of it

4. Approach Overview • Sea of cameras: produce multiple views of an object • A silhouette contour of a closed object is a circular function: can be made rotation-invariant • Multiple silhouettes can be used to improve the accuracy of a query (i.e. get me the object that has these two silhouettes)

5. Sea of cameras • The “sea” of cameras can be sampled over a hemisphere because silhouettes will be the same (mod flip) • Regular distribution on a sphere over samples… • Ideal case: random distribution of K camera points over a hemisphere that changes for each model • My version: custom placed locations of 20 points on a hemisphere that is constant across all shapes

6. Contour extraction (1) • Most practical shapes will have: • Sampling artifacts in the silhouette image • Interior contours (i.e. no one dominant contour) silhouette edges

7. Contour extraction (2) • Perform morphological closing of silhouette image with a disk structuring element • Use a laplacian-of-gaussian edge detector to extract contour pixels

8. Contour extraction (3) • Using a maze walking algorithm, find a closed loop through the contour pixels to produce a single closed group of pixels • This is done for all connected components • I bet we could use all of the components to do partial matching on this shape • … but for now, we chose the largest contour that results

9. Silhouette Parameterization • Goals: rotation, scale, translation invariance • RDF (radial distance function) • f(c) = r where r is maximal object extent in direction c from centroid • As in spherical harmonics • Arc length parameterization: • f(c) = r + k(s)i parameterization about centroid • Where k(s) is the stretch required to fit the curve to the surface

10. Matching Metric • Let dist(s1, s2) be the sum of squared distances between the descriptor for two contours (“samples”) • Option 1: best sample from query and model • Let dist(mi,q1…K) for some model mi and K samples q1…j be defined as • The sample sd in model mi that is closest to any qi • Option 2: all samples in query, best in model • Sum over all k of qi • The sample sd in model mi that is closest to qi

11. System Details (1) • I start with Princeton Shape Benchmark geometry and classifications • Silhouette sea is generated by a glut app in Linux using depth buffer readback • Images are copied to a machine with Matlab where morphological closing and edge detection is done • … and then copied back… grr… • Useful stat: 622 models yields 1.5 GB of images, but only a 7.5mb database

12. System Details (2) • Edge images are processed by a maze search algorithm to find the largest closed contour • Contours are sampled evenly to 200 points using Gnu Scientific Library • The parameterization is FFT’d with FFTw • The resulting descriptors are stored in a binary database • We can set an epsilon value that allows the same sample to be appear in multiple models if it is similar enough (compression!) • Precision vs recall is generated internally by the database

13. Control tests: rotation invariance • Match is somewhat sensitive to noise… e.g. higher frequency noise introduced by the sampling of the contours

14. Control tests: rotation invariance • Tinkering with filtering and other stuff doesn’t help much… using the RDF does.

15. Control tests: Scale invariance

16. Circular parameterization andModel match option 1 20 classes shown57 total classes622 models

17. Circular parameterization andModel match option 1 20 classes shown57 total classes622 models

18. Circular parameterization andModel match option 1 20 classes shown57 total classes622 models

19. Radial distance parameterization andModel match option 1 15 classes shown57 total classes622 models 1 silhouette used in query

20. Radial distance parameterization andModel match option 1 15 classes shown57 total classes622 models 2 silhouettes used in query

21. Radial distance parameterization andModel match option 1 15 classes shown57 total classes622 models 8 silhouettes used in query

22. Radial distance parameterization andModel match option 2 1 silhouette used in query 15 classes shown57 total classes622 models

23. Radial distance parameterization andModel match option 2 4 silhouettes used in query 15 classes shown57 total classes622 models

24. Radial distance parameterization andModel match option 2 4 silhouettes used in query 15 classes shown57 total classes622 models

25. Radial distance parameterization andModel match option 2 8 silhouettes used in query15 classes shown57 total classes622 models

26. Radial distance parameterization andModel match option 2 8 silhouettes used in query, 57 classes shown57 total classes622 models