Richard Baraniuk Kevin Kelly Rice University

1. Compressive Optical Imaging Systems – Theory, Devices, Implementation Richard Baraniuk Kevin Kelly Rice University David Brady Rebecca Willett Duke University Rice/Duke | Compressive Optical Devices | August 2007

2. Project OverviewRichard Baraniuk Rice/Duke | Compressive Optical Devices | August 2007

3. Digital Revolution

4. camera arrays hyperspectral cameras distributed camera networks

5. Sensing by Sampling • Long-established paradigm for digital data acquisition • sampledata at Nyquist rate (2x bandwidth) • compressdata (signal-dependent, nonlinear) • brick wall to resolution/performance sample compress transmit/store sparsewavelettransform receive decompress

6. Compressive Sensing (CS) • Directly acquire “compressed” data • Replace samples by more general “measurements” compressive sensing transmit/store receive reconstruct

7. Compressive Sensing • When data is sparse/compressible, can directly acquire a condensed representation with no/little information lossthrough dimensionality reduction sparsesignal measurements sparsein somebasis

8. Compressive Sensing • When data is sparse/compressible, can directly acquire a condensed representation with no/little information loss • Random projection will work sparsesignal measurements sparsein somebasis for signal reconstruction [Candes-Romberg-Tao, Donoho, 2004]

9. Compressive Optical Imaging Systems –Theory, Devices, and Implementation • $400k budget for roughly April 2006-2007 • administered by ONR • Rice portion expended; Duke portion in NCE • Goals: • forge collaboration between Rice and Duke teams • demonstrate new Compressive Imaging technologies • hardware testbeds/demos at Rice and Duke • new theory/algorithms • quantify performance • articulate emerging directions • Collaborations: • telecons, visits, joint projects, joint papers, artwork

10. Gerhard Richter 4096 Farben / 4096 Colours 1974254 cm X 254 cmLaquer on CanvasCatalogue Raisonné: 359 Museum Collection:Staatliche Kunstsammlungen Dresden (on loan) Sales history: 11 May 2004Christie's New York Post-War and Contemporary Art (Evening Sale), Lot 34US$3,703,500   Rice/Duke | Compressive Optical Devices | August 2007

11. Gerhard Richter Dresden CathedralStained Glass Rice/Duke | Compressive Optical Devices | August 2007

12. Agenda • Rebecca Willett, Duke [theory/algorithms] • Kevin Kelly, Rice [hardware] • David Brady, Duke [hardware] • Richard Baraniuk, Rice [theory/algorithms] • Discussion and Conclusions

13. Compressive Image ProcessingRichard Baraniuk Rice/Duke | Compressive Optical Devices | August 2007

14. Matthew Moravec Mona Sheikh Jason Laska Mike WakinMarco DuarteMark DavenportShri SarvothamPetrosBoufounos

15. Image Classification/Segmentationusing Duke Hyperspectral System(with Rebecca Willett) Rice/Duke | Compressive Optical Devices | August 2007

16. Information Scalability • If we can reconstruct a signal from compressive measurements, then we should be able to perform other kinds of statistical signal processing: • detection • classification • estimation … • Hyperspectral image classification/segmentation

17. Classification Example spectrum 2 spectrum 1 spectrum 3

18. Nearest Projected Neighbor • normalize measurements • compute nearest neighbor

19. Naïve Results block size 32 8 16

20. Results naïve independent classification tree-based classification

21. Voting / Cycle Spinning block radius in pixels 16 20 24 28 32

22. Summary • Direct hyperspectral classification/segmentation without reconstructing 3D data cube • Future directions • replace nearest projected neighbor with more sophisticated methods • smashed filter • projected SVM • quad-tree based multiscale segmentation (HMTseg, …) • Joint paper in the works

23. Performance Analysis of Multiplexed Cameras Rice/Duke | Compressive Optical Devices | August 2007

24. Single-Pixel Camera Analysis • Analyze performance in terms of • dynamic range and #bits of A/D • MSE due to photon counting noise • number of measurements photon detector imagereconstructionorprocessing DMD DMD random pattern on DMD array

25. Single Pixel Image Acquisiton For a N-pixel, K-sparse image under T-second exposure: Raster Scan: Acquire one pixel at a time, repeat N times Basis Scan: Acquire one coefficient of image in a fixed basis at a time, repeat N times CS Scan: Acquire one incoherent/random projection of the image at a time, repeat times Rice/Duke | Compressive Optical Devices | August 2007

26. Worst-Case Performance N: Number of pixels P: Number of photons per pixel T: Total capture time M: Number of measurements CN: CS noise amplification constant Sensor array shown as baseline Table shows requirements to match worst-case performance CS beats Basis Scan if Rice/Duke | Compressive Optical Devices | August 2007

27. Single Pixel Camera Experimental Performance N = 16384 M = 1640  = Daub-8 Rice/Duke | Compressive Optical Devices | August 2007

28. Multiplexed Camera Analysis Dude, you gotta multiplex! S photon detectors lens(es) imagereconstructionorprocessing DMD DMD random pattern on DMD array

29. S-Pixel Camera Performance N: Number of pixels P: Number of photons per pixel T: Total capture time M: Number of measurements CN: CS noise amplification constant Sensor array shown as baseline M measurements split across S sensors Single pixel camera: S = 1 Rice/Duke | Compressive Optical Devices | August 2007

30. S-Pixel Camera Performance N: Number of pixels P: Number of photons per pixel T: Total capture time M: Number of measurements CN: CS noise amplification constant Sensor array shown as baseline M measurements split across S sensors Single pixel camera: S = 1 CS beats Basis Scan if Rice/Duke | Compressive Optical Devices | August 2007

31. Smashed Filter –Compressive Matched Filtering Rice/Duke | Compressive Optical Devices | August 2007

32. Information Scalability • If we can reconstruct a signal from compressive measurements, then we should be able to perform other kinds of statistical signal processing: • detection • classification • estimation … • Smashed filter: compressive matched filter

33. Matched Filter • Signal classification in additive white Gaussian noise • LRT: classify test signal as from Class iif it is closest to template signal i • GLRT: when test signal can be a transformed version of template, use matched filter • When signal transformations are well-behaved, transformed templates form low-dimensional manifolds • GLRT = matched filter= nearest manifold classification M1 M3 M2

34. Compressive LRT • Compressive observations • By the Johnson-Lindenstrauss Lemma, random projection preserves pairwise distances with high probability

35. Smashed Filter • Compressive observations of transformed signal • Theorem: Structure of smooth manifolds is preserved by random projection w.h.p. provided distances, geodesic distances, angles, volume, dimensionality, topology, local neighborhoods, … [Wakin et al 2006; to appear in Foundations on Computational Mathematics] M1 M1 M3 M2 M3 M2

36. Stable Manifold Embedding Theorem: Let F ½ RN be a compact K-dimensional manifold with • condition number 1/t (curvature, self-avoiding) • volume V Let F be a random MxN orthoprojector with Then with probability at least 1-r, the following statement holds: For every pair x,y2F • [Wakin et al 2006]

37. Manifold Learning from Compressive Measurements LaplacianEigenmaps ISOMAP HLLE R4096 RM M=15 M=15 M=20

38. Smashed Filter – Experiments • 3 image classes: tank, school bus, SUV • N = 65536 pixels • Imaged using single-pixel CS camera with • unknown shift • unknown rotation

39. Smashed Filter – Unknown Position • Object shifted at random (K=2 manifold) • Noise added to measurements • Goal: identify most likely position for each image class identify most likely class using nearest-neighbor test more noise classification rate (%) avg. shift estimate error more noise number of measurements M number of measurements M

40. Smashed Filter – Unknown Rotation • Object rotated each 2 degrees • Goals: identify most likely rotation for each image class identify most likely class using nearest-neighbor test • Perfect classification with as few as 6 measurements • Good estimates of rotation with under 10 measurements avg. rot. est. error number of measurements M

41. How Low Can M Go? • Empirical evidence that many fewer than measurements are needed for effective classification • Late-breaking results (experimental+nascent theory)

42. Summary – Smashed Filter • Compressive measurementsare info scalable reconstruction > estimation > classification > detection • Random projections preserve structure of smooth manifolds (analogous to sparse signals) • Smashed filter: dimension-reduced GLRT for parametrically transformed signals • exploits compressive measurements and manifold structure • broadly applicable: targets do not have to have sparse representation in any basis • effective for detection/classification • number of measurements required appears to be independent of the ambient dimension

43. Compressive Phase Retrievalfor Fourier Imagers Rice/Duke | Compressive Optical Devices | August 2007

44. Coherent Diffraction Imaging • Image by sampling in Fourier domain • Challenge: we observe only the magnitude of the Fourier measurements

45. Phase Retrieval • Given: Fourier magnitude + additional constraints (typically support) • Goal: Estimate phase of Fourier transform • Compressive Phase Retrieval (CPR) replace image support constraint with a sparsity/compressibility constraintnonconvex reconstruction

47. Conclusions and Future Directions Rice/Duke | Compressive Optical Devices | August 2007

48. Project Outcomes • Forged collaboration between Rice and Duke teams • several joint papers in progress • Demonstrated new Compressive Imaging technologies • hardware testbeds/demos • hyperspectral, low-light, infrared DMD cameras • coded aperture spectral imagers • new theory/algorithms • spectral image reconstruction/classification methods • smashed filter • Quantified performance • coded aperture tradeoffs • multiplexing tradeoff • number of measurements required for reconstruction/classification Rice/Duke | Compressive Optical Devices | August 2007

49. Emerging Directions • Nonimaging cameras • exploit information scalability • attentive/adaptive cameras • meta-analysis • separating “imaging process” from “display” • Multiple cameras • image beamforming, 3D geometry imaging, … • Deeper links between physics and signal processing • significance of coherence and spectral projections • Links to analog-to-information program • nonidealities as challenges vs. opportunities • Other modalities • THz, LWIR/MWIR, UV, soft x-rays, …