Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012

1. Short-course Compressive Sensing of Videos Venue CVPR 2012, Providence, RI, USA June 16, 2012 Organizers: Richard G. Baraniuk Mohit Gupta Aswin C. Sankaranarayanan Ashok Veeraraghavan

2. Part 2: Compressive sensingMotivation, theory, recovery • Linear inverse problems • Sensing visual signals • Compressive sensing • Theory • Hallmark • Recovery algorithms • Model-based compressive sensing • Models specific to visual signals

3. Linear inverse problems

4. Linear inverse problems • Many classic problems in computer can be posed as linear inverse problems • Notation • Signal of interest • Observations • Measurement model • Problem definition: given , recover measurement matrix measurement noise

5. Linear inverse problems • Problem definition: given , recover • Scenario 1 • We can invert the system of equations • Focus more on robustness to noise via signal priors

6. Linear inverse problems • Problem definition: given , recover • Scenario 2 • Measurement matrix has a (N-M) dimensional null-space • Solution is no longer unique • Many interesting vision problem fall under this scenario • Key quantity of concern: Under-sampling ratio M/N

7. Image super-resolution Low resolution input/observation 128x128 pixels

8. Image super-resolution 2x super-resolution

9. Image super-resolution 4x super-resolution

10. Image super-resolution Super-resolution factor D Under-sampling factor M/N = 1/D2 General rule: The smaller the under-sampling, the more the unknowns and hence, the harder the super-resolution problem

11. Many other vision problems… • Affine rank minimizaiton, Matrix completion, deconvolution, Robust PCA • Image synthesis • Infilling, denoising, etc. • Light transport • Reflectance fields, BRDFs, Direct global separation, Light transport matrices • Sensing

12. Sensing visual signals

13. High-dimensional visual signals Fog Volumetric scattering Reflection Human skin Sub-surface scattering Refraction Electron microscopy Tomography

14. The plenopticfunction Collection of allvariationsof light in a scene space (3D) spectrum (1D) time (1D) angle (2D) Different slices reveal different scene properties Adelson and Bergen (91)

15. The plenopticfunction space (3D) time (1D) spectrum (1D) angle (2D) High-speed cameras Hyper-spectral imaging Lytro light-field camera

16. Sensing the plenoptic function • High-dimensional • 1000 samples/dim == 10^21 dimensional signal • Greater than all the storage in the world • Traditional theories of sensing fail us!

17. Resolution trade-off • Key enabling factor: Spatial resolution is cheap! • Commercial cameras have 10s of megapixels • One idea is the we trade-off spatial resolution for resolution in some other axis

18. Spatio-angular tradeoff [Ng, 2005]

19. Spatio-angular tradeoff [Levoy et al. 2006]

20. Spatio-temporal tradeoff [Bub et al., 2010] Stagger pixel-shutter within each exposure

21. Spatio-temporal tradeoff [Bub et al., 2010] Rearrange to get high temporal resolution video at lower spatial-resolution

22. Resolution trade-off • Very powerful and simple idea • Drawbacks • Does not extend to non-visible spectrum • 1 Megapixel SWIR camera costs 50-100k • Linear and global tradeoffs • With today’s technology, cannot obtain more than 10x for video without sacrificing spatial resolution completely

23. Compressive sensing

24. Sense by Sampling sample

25. Sense by Sampling too much data! sample

26. Sense then Compress sample compress JPEG JPEG2000 … decompress

27. Sparsity largewaveletcoefficients (blue = 0) pixels

28. Sparsity largewaveletcoefficients (blue = 0) largeGabor (TF)coefficients pixels widebandsignalsamples frequency time

29. Concise Signal Structure • Sparse signal: only K out of N coordinates nonzero sorted index

30. Concise Signal Structure • Sparse signal: only K out of N coordinates nonzero • model: union of K-dimensional subspaces aligned w/ coordinate axes sorted index

31. Concise Signal Structure • Sparse signal: only K out of N coordinates nonzero • model: union of K-dimensional subspaces • Compressible signal: sorted coordinates decay rapidly with power-law power-lawdecay sorted index

32. What’s Wrong with this Picture? • Why go to all the work to acquire N samples only to discard all but K pieces of data? sample compress decompress

33. What’s Wrong with this Picture? • nonlinear processing • nonlinear signal model • (union of subspaces) linear processing linear signal model (bandlimited subspace) sample compress decompress

34. Compressive Sensing • Directly acquire “compressed” data via dimensionality reduction • Replace samples by more general “measurements” compressive sensing recover

35. Sampling • Signal is -sparse in basis/dictionary • WLOG assume sparse in space domain sparsesignal nonzeroentries

36. Sampling • Signal is -sparse in basis/dictionary • WLOG assume sparse in space domain • Sampling sparsesignal measurements nonzeroentries

37. Compressive Sampling • When data is sparse/compressible, can directly acquire a condensed representation with no/little information loss through linear dimensionality reduction sparsesignal measurements nonzero entries

38. How Can It Work? • Projection not full rank…… and so loses information in general • Ex: Infinitely many ’s map to the same(null space)

39. How Can It Work? • Projection not full rank…… and so loses information in general • But we are only interested in sparse vectors columns

40. How Can It Work? • Projection not full rank…… and so loses information in general • But we are only interested in sparse vectors • is effectively MxK columns

41. How Can It Work? • Projection not full rank…… and so loses information in general • But we are only interested in sparse vectors • Design so that each of its MxK submatrices are full rank (ideally close to orthobasis) • Restricted Isometry Property (RIP) columns

42. Restricted Isometry Property (RIP) Preserve the structure of sparse/compressible signals K-dim subspaces

43. Restricted Isometry Property (RIP) K-dim subspaces • “Stable embedding” • RIP of order 2K implies: for all K-sparse x1 and x2

44. RIP = Stable Embedding An information preserving projection preserves the geometry of the set of sparse signals RIP ensures that

45. How Can It Work? • Projection not full rank…… and so loses information in general • Design so that each of its MxK submatrices are full rank (RIP) • Unfortunately, a combinatorial, NP-complete design problem columns

46. Insight from the 70’s [Kashin, Gluskin] • Draw at random • iid Gaussian • iid Bernoulli … • Then has the RIP with high probability provided columns

47. Randomized Sensing • Measurements = random linear combinationsof the entries of the signal • No information loss for sparse vectors whp sparsesignal measurements nonzero entries

48. CS Signal Recovery • Goal: Recover signal from measurements • Problem: Randomprojection not full rank(ill-posed inverse problem) • Solution: Exploit the sparse/compressiblegeometry of acquired signal

49. CS Signal Recovery • Random projection not full rank • Recovery problem:givenfind • Null space • Search in null space for the “best”according to some criterion • ex: least squares (N-M)-dim hyperplaneat random angle

50. Signal Recovery • Recovery: given(ill-posed inverse problem) find (sparse) • Optimization: • Closed-form solution: