Synthetic aperture confocal imaging

1. Synthetic aperture confocal imaging Marc Levoy Billy Chen Vaibhav Vaish Mark Horowitz Ian McDowall Mark Bolas

2. technologies large camera arrays large projector arrays camera–projector arrays optical effects synthetic aperture photography synthetic aperture illumination synthetic confocal imaging applications • partially occluding environments • weakly scattering media examples foliage murky water Outline

3. Stanford Multi-Camera Array[Wilburn 2002] • 640 × 480 pixels ×30fps × 128 cameras • synchronized timing • continuous video streaming • flexible physical arrangement

4. Ways to use large camera arrays • widely spaced light field capture • tightly packed high-performance imaging • intermediate spacing synthetic aperture photography

5. Synthetic aperture photography

6. Synthetic aperture photography

7. Synthetic aperture photography

8.  Synthetic aperture photography

9.  Synthetic aperture photography

10.  Synthetic aperture photography

11. Related work • not like synthetic aperture radar (SAR) • more like X-ray tomosynthesis • [Levoy and Hanrahan, 1996] • [Isaksen, McMillan, Gortler, 2000]

12. Example using 45 cameras

13. Synthetic pull-focus

14. Crowd scene

15. Crowd scene

16. Synthetic aperture photographyusing an array of mirrors • 11-megapixel camera • 22 planar mirrors ?

19. Synthetic aperture illumation

20. Synthetic aperture illumation • technologies • array of projectors • array of microprojectors • single projector + array of mirrors • applications • bright display • autostereoscopic display [Matusik 2004] • confocal imaging [this paper]

21. light source pinhole Confocal scanning microscopy

22. r light source pinhole pinhole photocell Confocal scanning microscopy

23. Confocal scanning microscopy light source pinhole pinhole photocell

24. Confocal scanning microscopy light source pinhole pinhole photocell

25. [UMIC SUNY/Stonybrook]

26. → 5 beams → 0 or 1 beam Synthetic confocal scanning light source

27. → 5 beams → 0 or 1 beam Synthetic confocal scanning light source

28. d.o.f. → 5 beams → 0 or 1 beam Synthetic confocal scanning • works with any number of projectors ≥ 2 • discrimination degrades if point to left of • no discrimination for points to left of • slow! • poor light efficiency

29. Synthetic coded-apertureconfocal imaging • different from coded aperture imaging in astronomy • [Wilson, Confocal Microscopy by Aperture Correlation, 1996]

30. Synthetic coded-apertureconfocal imaging

31. Synthetic coded-apertureconfocal imaging

32. Synthetic coded-apertureconfocal imaging

33. Synthetic coded-apertureconfocal imaging 100 trials → 2 beams × ~50/100 trials ≈ 1 → ~1 beam × ~50/100 trials ≈ 0.5

34. Synthetic coded-apertureconfocal imaging 100 trials → 2 beams × ~50/100 trials ≈ 1 → ~1 beam × ~50/100 trials ≈ 0.5 floodlit → 2 beams → 2 beams trials – ¼ × floodlit → 1 – ¼ ( 2 ) ≈ 0.5 → 0.5 – ¼ ( 2 ) ≈ 0

35. Synthetic coded-apertureconfocal imaging 100 trials → 2 beams × ~50/100 trials ≈ 1 → ~1 beam × ~50/100 trials ≈ 0.5 floodlit → 2 beams → 2 beams trials – ¼ × floodlit → 1 – ¼ ( 2 ) ≈ 0.5 → 0.5 – ¼ ( 2 ) ≈ 0 • 50% light efficiency • any number of projectors ≥ 2 • no discrimination to left of • works with relatively few trials (~16)

36. Synthetic coded-apertureconfocal imaging 100 trials → 2 beams × ~50/100 trials ≈ 1 → ~1 beam × ~50/100 trials ≈ 0.5 floodlit → 2 beams → 2 beams trials – ¼ × floodlit → 1 – ¼ ( 2 ) ≈ 0.5 → 0.5 – ¼ ( 2 ) ≈ 0 • 50% light efficiency • any number of projectors ≥ 2 • no discrimination to left of • works with relatively few trials (~16) • needs patterns in which illumination of tiles are uncorrelated

37. Example pattern

38. Patterns with less aliasing

39. Implementation using an array of mirrors

41. Confocal imaging in scattering media • small tank • too short for attenuation • lit by internal reflections

42. Experiments in a large water tank 50-foot flume at Wood’s Hole Oceanographic Institution (WHOI)

43. Experiments in a large water tank • 4-foot viewing distance to target • surfaces blackened to kill reflections • titanium dioxide in filtered water • transmissometer to measure turbidity

44. Experiments in a large water tank • stray light limits performance • one projector suffices if no occluders

45. Seeing through turbid water floodlit scanned tile

46. [Ballard/IFE 2004] Application tounderwater exploration [Ballard/IFE 2004]

47. Research challenges in SAP and SAI • theory • aperture shapes and sampling patterns • illumination patterns for confocal imaging • optical design • How to arrange cameras, projectors,lenses, mirrors, and other optical elements? • How to compare the performance of different arrangements (in foliage, underwater,...)?

48. Challenges (continued) • systems design • multi-camera or multi-projector chips • communication in camera-projector networks • calibration in long-range or mobile settings • algorithms • tracking and stabilization of moving objects • compression of dense multi-view imagery • shape from light fields

49. Challenges (continued) • applications of confocal imaging • remote sensing and surveillance • shape measurement • scientific imaging • applications of shaped illumination • shaped searchlights for surveillance • shaped headlamps for driving in bad weather • selective lighting of characters for stage and screen

50. Computational imagingin other fields • medical imaging • rebinning • tomography • airborne sensing • multi-perspective panoramas • synthetic aperture radar • astronomy • coded-aperture imaging • multi-telescope imaging