Compressed Sensing Theory Geometric Interpretations

Compressed Sensing Theory Geometric Interpretations

Télécharger la présentation

Compressed Sensing Theory Geometric Interpretations

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

1. Compressed Sensing TheoryGeometric Interpretations April 7th 2016, Ilmenau Prof. Giovanni Del Galdo

2. Compressed Sensing TheoryGeometric Interpretations April 7th 2016, Ilmenau Prof. Giovanni Del Galdo With contributions from… Dr. Florian RömerAnastasia LavrenkoAlexandra CraciunMagdalena PrusMohamed Gamal IbrahimRoman Alieiev Heavily inspired by… the tutorial of Dr. Dejan E. Lazichgiven at the 22nd meeting of the ITG section on “Applied Information Theory” on Oct. 7th, 2013

3. Goals of this lecture • Answer the following questions: • Why is the desired sparse solution unique? • Why is the hyperplane of solutions tangent to the L1-ball at the right point? • What is the impact of measurement noise? ?

4. Complete Data Model Reconstructionstrategy

5. Solution Set for the Equality Constraint • x must lie on a line

6. Lp-Norms and Lp-Balls • Definition of Lp-norm: • Lp-Balls for N = 2

7. Minimization Problem: p=0.5

8. Minimization Problem: p=0.5 L2 2D animation

9. Minimization Problem: p=0.5 L2 2D animation • The “arms” of the L0.5 ball reach out making sparse solutions favored wrt non-sparse

10. Minimization Problem: p=1 • Same effect as L0.5, although not as prominent

11. Minimization Problem: p=1

12. Minimization Problem: p=1

13. Minimization Problem: p=2

14. Minimization Problem: p=2

15. Minimization Problem: p=2 • This is the LS solution

16. Escape Velocities for the L1-ball • Vertices move faster than edges, which move faster than sides • They correspond to 1-sparse, 2-sparse, and 3-sparse respectively

17. Choice of p-Norm • p>1 lead to non-sparse solutions • p<=1 lead to sparse solutions • p=0 = cardinality • logical choice • gives the sparsest solution • NP-hard • p=1 • identical to p=0 for practical cases • linear and convex problem

18. Goals of this lecture • Answer the following questions: • Why is the desired sparse solution unique? • Why is the hyperplane of solutions tangent to the L1-ball at the right point? • What is the impact of measurement noise? ?

19. Outline • General geometrical considerations • affine subspace of solutions for a generalized System of Linear Equations (SLE) • subspaces spanned by sparse vectors • intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i.e., When does an SLE have sparse solutions? • Geometrical considerations specific to Compressed Sensing • solutions for a system originating from a compressed sensing scenario • impact of the noise: designing the sensing matrix • Conclusions

20. System of Linear Equations (SLE) underdetermined overdetermined fully determined infinite number ofsolutions one unique solution no solutions ! assuming B to be full rank

21. Solutions to the i-th Linear Equation solution to the homogeneous problem solution to the inhomogeneous problem

22. Solutions to the Homogeneous Linear Equation • Any point x0 on the plane orthogonalto biis a solution to the homogeneous linearequation

23. Solutions to the Inhomogeneous Linear Equation • Distance between the planes: • Any point on the affine planeis a solution to the inhomogeneous linearequation

24. Solutions to the Inhomogeneous SLE • Each equation removes a degreeof freedom • There exists one unique solution

25. Shooting for Higher Dimensional Spaces • In order to predict the characteristics of the affine subspace of solutions we cannot rely on intuition alone • Therefore, we need to introduce more powerful tools • the definition of affine subspaces • the rules applying to the intersection of affine subspaces ND 3D

26. K-Flats, Affine Subspaces in N-dimensional Ambient Space • The origin-free generalization of points, lines, planes, and K-dimensional subspaces are termed: • affine subspaces of dimension 0, 1, 2, and K, respectively or • 0-flats, 1-flats, 2-flats, and K-flats, respectively • A K-flat is determined by K+1 linearly independent points • K points are linearly independent iff • taking a subset, no 3 points lie on a line (1-flat) • taking a subset, no 4 points lie on a plane (2-flat) • … • taking a subset, no k points lie on a (k-2)-flat, for 2<=k<=K • Ambient space of dimension N: the N-flat which contains the universeconsidered

27. K-Flats, Affine Subspaces in N-dimensional Ambient Space • A line is a 1-flat and is determined by 2 linearly independent points • A plane is a 2-flat and is determined by 3 linearly independent points • 3 points are linearly independent if no 2 points overlap and no 3 points lie on a line

28. Intersection of Affine Subspaces • Assume a p-flat and a q-flat in N-dimensional ambient space • Ignoring special cases, the intersection of the two flats is • (p+q-N)-flat, if p+q-N >= 0 • empty, if p+q-N < 0 • If two flats obey the rules above, they are said to be in • General Position • Randomly generated subspaces are in General Position with probability one

29. Intersection of Affine Subspaces: Examples Two lines in 3D space p = 1 q = 1 N = 3 Intersection = empty …as 1 + 1 – 3 = -1 The lines are skew

30. Intersection of Affine Subspaces: Examples Two lines in 2D space p = 1 q = 1 N = 2 Intersection = 0-flat …as 1 +1 – 2 = 0 The lines intersect in a point Degeneratecase: Twooverlappinglines Intersection = 1-flat

31. Intersection of Affine Subspaces: Examples Two planes in 3D space p = 2 q = 2 N = 3 Intersection = 1-flat …as 2 + 2 – 3 = 1 The planes intersect in a line

32. Intersection of Affine Subspaces: Examples A special case can be visualized! The rest of the plane is in the fourth dimension Two planes in 4D space p = 2 q = 2 N = 4 Intersection = 0-flat, i.e., a point …as 2 + 2 – 4 = 0 The two planes intersect in a point

33. Degenerate Cases: Examples • Two 1-flats (lines) overlapping in 2D • The General Position would be an intersection in one point • Two 2-flats (planes) intersecting on a line in 5D • The General Position would be an empty intersection • Affine subspaces generated randomly are degenerate with probability zero

34. Flats, their Intersection for a SLE • Each linear constraint gives rise to a (N-1)-flat • The subspace of solutions is the intersection of the M (N-1)-flats in N-dimensional space. It is an (N-M)-flat!

35. Flats, their Intersection for a SLE

36. Recap: Geometrical Interpretation on the SLE • Given a SLE featuring • an N dimensional unknown vector x • M linear equations • The solutions to each linear equation span an affine subspace of dimensionality N-1, i.e., an (N-1)-flat • The solutions to the whole SLE span the intersection of the M (N-1)-flats • Given an underdetermined system of equations • the affine subspace of solutions has size N-M • we call it: the “Solutions Monster” (N-M)-flat Solutions Monster

37. Outline • General geometrical considerations • affine subspace of solutions for a generalized System of Linear Equations (SLE) • subspaces spanned by sparse vectors • intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i.e., When does an SLE have sparse solutions? • Geometrical considerations specific to Compressed Sensing • solutions for a system originating from a compressed sensing scenario • impact of the noise: designing the sensing matrix • Conclusions

38. Affine Subspaces for K-sparse Vectors • The K-sparse vectors span specific affine subspaces in N-dimensional space • 1-sparse vectors span 1-flats

39. Affine Subspaces for K-sparse Vectors • 2-sparse vectors span 2-flats

40. Outline • General geometrical considerations • affine subspace of solutions for a generalized System of Linear Equations (SLE) • subspaces spanned by sparse vectors • intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i.e., When does an SLE have sparse solutions? • Geometrical considerations specific to Compressed Sensing • solutions for a system originating from a compressed sensing scenario • impact of the noise: designing the sensing matrix • Conclusions

41. Affine Subspaces for K-sparse Vectors • In general: • K-sparse vectors span K-flats • Given a system of linear equations in General Position: • What is the probability of having sparse solutions? I.e., when will the solutions monster touch the flats containing sparse vectors? ? 1-flat (N-M)-flat Solutions Monster

42. Intersection: Solution Monster <-> Sparse Flats • Assuming a SLE of M linear constraints in N-dimensional space • Assuming General Position the results in the table occur with Prob. = 1

43. Intersection: Solution Monster <-> Sparse Flats 2-flat 2-flat 1-flat 1-flat (N-M)-flat Solutions to the SLE 2-flat 1-flat • For K < M the Solutions Monster does not intersect with any K-flat carrying K-sparse solutions

44. Recap: Solution Monster vs. Sparse Flats • In the General Position, the Solutions Monster dodges all sparse vectors up to K = M-1 Me too! (N-M)-flat Solutions to the SLE

45. Outline • General geometrical considerations • affine subspace of solutions for a generalized System of Linear Equations (SLE) • subspaces spanned by sparse vectors • intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i.e., When does an SLE have sparse solutions? • Geometrical considerations specific to Compressed Sensing • solutions for a system originating from a compressed sensing scenario • impact of the noise: designing the sensing matrix • Conclusions

46. SLE in a Compressed Sensing Scenario • Let us now take an underdetermined system of equations originating from a sparse solution x • The Solution Monster corresponding to the homogeneous system will bein General Position (N-M)-flat Solutions Monster