Structured Sparse Principal Component Analysis

1. Structured Sparse Principal Component Analysis Authors: RodolpheJenatton, Guillaume Obozinski, Francis Bach Reading Group Presenter: Peng Zhang Cognitive Radio Institute Friday, October 01, 2010

2. Outline • Introduction (in Imaging Sense) • Principal Component Analysis (PCA) • Sparse PCA (SPCA) • Structured Sparse PCA (SSPCA) • Problem Statement • The SSPCA Algorithm • Experiments • Conclusion and Other Thoughts

3. Introduction (Imaging Sense) • The face recognition problem • A database includes a huge amount of faces • How to let computer to recognize different faces with database • The challenge • Huge amount of data • Computation complexity • The trick • Represent the face using a weighted “face dictionary” • Similar to code book in data compression • Example: An 200 X 200 pixel face can be represented by 100 coefficients using the “face dictionary” • The solution • Principal component analysis (PCA)

4. PCA • PCA • A compression method • Given a large amount of sample vectors {x} • 2nd moment statistics of the sample vectors • Eigen-decomposition finds the “dictionary” and “energy” of the dictionary codes • Eigen-vectors {v} form the “dictionary” • Eigen-values {d} give the “energy” of “dictionary” elements

5. PCA • Original signal can be represented using only part of the dictionary • Data is compressed with fewer elements • Meaning of “dictionary” v: • It is the weights of each elements in x • The problem for PCA for face recognition: No physical meaning for “dictionary”

6. PCA The Face Samples The “dictionary”, eigen-faces PCA These eigen-faces can reconstruct original faces perfectly, but make no sense in real life Face recognition

7. Structured SPCA Non-sparse Eigen-faces from PCA Sparse Eigen-faces from SPCA But the eigen-faces are still meaningless most of time • The SPCA goal: • Make dictionary more interpretable • The “sparse” solution: Limit the number of nonzeros

8. Structured SPCA Eigen-faces from SSPCA • The new idea, SSPCA • Eigen-faces will be meaningful when some structured constraints are set • Meaningful areas in faces are constrained in “grids”

9. Structured SPCA • This paper’s contribution • Add the “structure” constraint to make the dictionary more meaningful • How the constraint works • Meaningful dictionary is more close to “true” dictionary • Meaningful dictionary is more robust against noise • Meaningful dictionary is more accurate in face recognition

10. Outline • Introduction • Principal Component Analysis (PCA) • Sparse PCA (SPCA) • Structured Sparse PCA (SSPCA) • Problem Statement • The SSPCA Algorithm • Experiments • Conclusion and Other Thoughts

11. Problem Statement • From SPCA to SSPCA • The optimization problem • X is sample matrix, U is coefficient matrix, V is dictionary • ||.|| and are different types of norms • The trick in SPCA • L1 norm force the dictionary to be a sparse solution

12. Problem Statement Structured SPCA, however, deal with a mixed l1/l2 minimization: Right now it’s hard for me to understand the G and d

13. Problem Statement • In short, the norm constraints have the following effects • Dictionary has some structures • All non-zeros in the dictionary will be confined inside a grid

14. Outline • Introduction • Principal Component Analysis (PCA) • Sparse PCA (SPCA) • Structured Sparse PCA (SSPCA) • Problem Statement • The SSPCA Algorithm • Experiments • Conclusion and Other Thoughts

15. The SSPCA Algorithm • Making the dictionary sparser • The norm, • The new SSPCA problem:

16. The SSPCA Algorithm Methods to solve a sequence of convex problems

17. Excerpt from Author’s slide Excerpt from author’s slide:

18. Excerpt from Author’s slide

19. Excerpt from Author’s slide

20. Excerpt from Author’s slide

21. Excerpt from Author’s slide

22. Excerpt from Author’s slide

23. Excerpt from Author’s slide

24. Excerpt from Author’s slide

25. Excerpt from Author’s slide

26. Outline • Introduction • Principal Component Analysis (PCA) • Sparse PCA (SPCA) • Structured Sparse PCA (SSPCA) • Problem Statement • The SSPCA Algorithm • Experiments • Conclusion and Other Thoughts

27. Conclusion and Other Thoughts • Conclusion • This paper shows how to use SSPCA • SSPCA gets better performance in denoising, face recognition and classification • Other thoughts • Usually, the meaningful dictionary in communication signals is Fourier dictionary • But Fourier dictionary may not fit some transient signals or time-variant signals • How to manipulate the G, d and norms to set constraints for our needs?

28. Thank yOu!