Michael Elad The Computer Science Department The Technion – Israel Institute of technology

1. * Joint work with Michal Aharon Freddy Bruckstein The K–SVDDesign of Dictionaries for Redundant and Sparse Representation of Signals * Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000, Israel SPIE – Wavelets XI San-Diego: August 2nd, 2005

2. Welcome to Sparseland Agenda • A Visit to Sparseland • Motivating redundancy & Sparsity • 2. The Quest for a Dictionary – Fundamentals • Common Approaches? • 3. The Quest for a Dictionary – Practice • Introducing the K-SVD • 4. Results • Preliminary results and applications NOTE: This lecture concentrates on the fundamentals of this work, and thus does not fully overlap the accompanying paper. The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

3. Every column in D (dictionary) is a prototype signal (Atom). N • The vector is generated randomly with few non-zeros in random locations and random values. N A sparse & random vector K A fixed Dictionary Generating Signals in Sparseland M The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

4. M Multiply by D Sparseland Signals Are Interesting • Simple:Every generated signal is built as a linear combination of fewatoms from our dictionaryD • Rich:A general model: the obtained signals are a special type mixture-of-Gaussians (or Laplacians). • Popular: Recent work on signal and image processing adopt this model and successfully deploys it to applications. •  Sparseland is here !? The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

5. ? noise A sparse & random vector Multiply by D • Is or even close? • Practical ways to get ? 4 Major Questions OUR FOCUS TODAY!! Signal Processing inSparseland M Recent results give optimistic answers to these questions • How effective? • How do we get D? The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

6. Agenda • A Visit to Sparseland • Motivating redundancy & Sparsity • 2. The Quest for a Dictionary – Fundamentals • Common approaches? • 3. The Quest for a Dictionary – Practice • Introducing the K-SVD • 4. Results • Preliminary results and applications The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

7. M Multiply by D Given a set of data examples, we assume the Sparseland model and ask Problem Setting The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

8. Choose D – Modeling Approach • Pros: • Build on existing methods, • Fast transforms, • Proven optimality for the model. • Cons: • Relation to Sparseland ? Linearity? • How to adapt to other signals? • Bad for “smaller” signal families. Replace the model with another, well-defined simple mathematical, model (e.g. images as piece-wise C2 smooth regions with C2 smooth edges) and fit a dictionary accordingly, based on existing methods. Examples: Curvelet[Candes & Donoho]Contourlet[Do & Vetterli]Bandlet[Mallat & Le-Pennec]and others ... The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

9. OUR APPROACH FOR TODAY Choose D – Training Approach • Pros: • Fits the data, • Design fits the true objectives, • Can be adapted to small families. • Cons: • No fast version, • Slow training, • Scalability issues? Train the dictionary directly based on the given examples, optimizing w.r.t. sparsity and other desired properties (normalized atoms, etc.). Examples: ML (HVS)[Field & Olshausen]MAP [Lewicki & Sejnowski]MOD[Engan et. al.] ICA-like [Kreutz-Delgado et. al.]and others ... The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

10. Agenda • A Visit to Sparseland • Motivating redundancy & Sparsity • 2. The Quest for a Dictionary – Fundamentals • Common Approaches? • 3. The Quest for a Dictionary – Practice • Introducing the K-SVD • 4. Results • Preliminary results and applications The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

11. D X  A Each example has a sparse representation with no more than L atoms Each example is a linear combination of atoms from D Practical Approach – Objective Field & Olshausen (96’) Engan et. al. (99’) Lewicki & Sejnowski (00’) Cotter et. al. (03’) Gribonval et. al. (04’) (n,K,L are assumed known) The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

12. D Initialize D Sparse Coding Nearest Neighbor XT Dictionary Update Column-by-Column by Mean computation over the relevant examples K–Means For Clustering Clustering: An extreme sparse coding The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

13. D Initialize D Sparse Coding Use MP or BP XT Dictionary Update Column-by-Column by SVD computation The K–SVD Algorithm – General Aharon, Elad, & Bruckstein (`04) The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

14. D D is known! For the jth item we solve XT K–SVD: Sparse Coding Stage Pursuit Problem !!! The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

15. Gk: The examples in and that use the column dk. Let us fix all A and D apart from the kth column and seek both dk and the kth column in A to better fit the residual! The content of dk influences only the examples in Gk. K–SVD: Dictionary Update Stage D The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

16. We should solve: dk is obtained by SVD on the examples’ residual in Gk. K–SVD: Dictionary Update Stage D ResidualE The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

17. Agenda • A Visit to Sparseland • Motivating redundancy & Sparsity • 2. The Quest for a Dictionary – Fundamentals • Common Approaches? • 3. The Quest for a Dictionary – Practice • Introducing the K-SVD • 4. Results • Preliminary results and applications The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

18. Create A 2030 random dictionary with normalized columns Generate 2000 signal examples with 3 atoms per each and add noise Train a dictionary using the KSVD and MOD and compare D D Results K–SVD: A Synthetic Experiment The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

19. Overcomplete Haar 10,000 sample 8-by-8 images. 441 dictionary elements. Approximation method: OMP K-SVD: K–SVD on Images The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

20. Given an image with missing values Get the recovered image Multiply the found representation by the complete dictionary Apply pursuit (per each block of size(88) using a decimated dictionary with rows removed 60% missing pixels K-SVD Results Average # coefficients 4.08 RMSE: 11.68 Haar Results Average # coefficients 4.42 RMSE: 21.52 Filling–In Missing Pixels The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

21. Summary • Today we discussed: • 1. A Visit to Sparseland • Motivating redundancy & Sparsity • 2. The Quest for a Dictionary – Fundamentals • Common Approaches? • 3. The Quest for a Dictionary – Practice • Introducing the K-SVD • 4. Results • Preliminary results and applications • Open Questions: • Scalability – treatment of bigger blocks and large images. • Uniqueness? Influence of noise? • Equivalence? A guarantee to get the perfect dictionary? • Choosing K? What forces govern the redundancy? • Other applications? … http://www.cs.technion.ac.il/~elad The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

22. Supplement Slides The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

23. M Multiply by D Uniqueness? Is D unique in explaining the origin of the signals? (n,K,L are assumed known) The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

24. M • “Rich Enough”: The signals from could be clustered to groups that share the same support. At least L+1 examples per each are needed. Uniqueness? YES !! If is rich enough* and if then D is unique. Uniqueness Aharon, Elad, & Bruckstein (`05) Comments: • This result is proved constructively, but the number of examples needed to pull this off is huge – we will show a far better method next. • A parallel result that takes into account noise could be constructed similarly. The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

25. Haar dictionary DCT dictionary K-SVD dictionary OMP with error bound Naïve Compression The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals

26. Naïve Compression DCT BPP = 1.01 RMSE = 8.14 K-SVD BPP = 0.502 RMSE = 7.67 Haar BPP = 0.784 RMSE = 8.41 The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals