Tuning Pruning in Sparse Non-negative Matrix Factorization

1. Tuning Pruning in Sparse Non-negative Matrix Factorization Morten Mørup DTU Informatics Intelligent Signal Processing Technical University of Denmark Joint work with Lars Kai Hansen DTU Informatics Intelligent Signal Processing Technical University of Denmark EUSIPCO’09 27 August 2009

2. Non-negative Matrix Factorization (NMF) V W H  VWH, V≥0,W≥0,H≥0 Nature 1999 Daniel D. Lee • Gives part-based representation(and as such also promotes sparse representations)(Lee and Seung, 1999) • Also named Positive Matrix Factorization (PMF) (Paatero and Tapper, 1994) • Popularized due to a simple algorithmic procedure based on multiplicative update(Lee & Seung, 2001) Sebastian Seung EUSIPCO’09 27 August 2009

3. Roadmap: Some important challenges in NMF • How to efficiently compute the decomposition (NMF is a non-convex problem) • How to resolve the non-uniqueness of the decomposition • How to determine the number of components (first part of this talk) A good starting point is not to use multiplicative updates Positive Orthant Convex Hull z z z NMF only unique when data adequately spans the positive orthant (Donoho & Stodden, 2004) y y y x x x (second part of this talk) We will demonstrate that Automatic Relevance Determination in Bayesian learning can address these challenges by tuning the pruning in sparse NMF EUSIPCO’09 27 August 2009

4. Multiplicative updates Step size parameter (Salakhutdinov, Roweis, Ghahramani, 2004) (Lee & Seung, 2001) EUSIPCO’09 27 August 2009

5. Other common approaches for solving the NMF problem • Active set procedure (Analytic closed form solution wihtin active set for LS-error)(Lawson and Hansen 1974), (R. Bro and S. de Jong 1997) • Projected gradient (C.-J. Lin 2007) MU do not converge to optimal solution!!!! EUSIPCO’09 27 August 2009

6. Sparseness has been imposed to alleviate the non-uniqueness of NMF (P. Hoyer 2002, 2004), (J. Eggert and E. Körner 2004) Sparseness motivated by the principle of parsimony, i.e. forming the simplest account. As such sparseness is also related to VARIMAX and ML-ICA based on sparse priors EUSIPCO’09 27 August 2009

7. Open problems for Sparse NMF (SNMF) • What is the adequate degree of sparsity imposed  • What is the adequate number of components K to model the data Both issues can be posed as the single problem of tuning the pruning in sparse NMF (SNMF). Hence, by imposing a component wise sparsity penalty the above problems boils down to determining k.k results in kth component turned off (i.e. removed). EUSIPCO’09 27 August 2009

8. Bayesian Learning and the Principle of Parsimony The explanation of any phenomenon should make as few assumptions as possible, eliminating those that make no difference in the observable predictions of the explanatory hypothesis or theory. William of Ockham To get the posterior probability distribution, multiply the prior probability distribution by the likelihood function and then normalize Bayesian learning embodies Occam’s razor, i.e. Complex models are penalized. The horizontal axis represents the space of possible data sets D. Bayes rule rewards models in proportion to how much they predicted the data that occurred. These predictions are quantified by a normalized probability distribution on D. Thomas Bayes David J.C. MacKay EUSIPCO’09 27 August 2009

9. SNMF in a Bayesian formulation Likelihoodfunction Prior (In the hierarchical Bayesian framework priors on  can further be imposed) EUSIPCO’09 27 August 2009

10. The log posterior for Sparse NMF is now given by The contribution in the log posterior from the normalization constant of the priors enables to learn from data the regularization strength  (This is also known as Automatic Relevance Determination (ARD)) When Inserting this value for  in the objective it can be seen that ARD corresponds to a reweighted L0-norm optimization scheme of the component activation EUSIPCO’09 27 August 2009

11. No closed form solution for posterior moments of W and H due to non-negativity constraint and use of non- conjugate priors. • Posterior distribution can be estimated by sampling approaches, c.f. previous talk by Mikkel Schmidt. • Point estimates of W and H can be obtained by maximum a posteriori (MAP) estimation forming a regular sparse NMF optimization problem. Tuning Pruning algorithmfor sparse NMF EUSIPCO’09 27 August 2009

12. Data results Handwritten digits: X256 Pixels x 7291 digits CBCL face database: X361 Pixels x 2429 faces Wavelet transformed EEG: X64 channels x 122976 tim.-freq. bins EUSIPCO’09 27 August 2009

13. Analyzing X vs. XT Handwritten digits (X): X256 Pixels x 7291 digits Handwritten digits (XT): X7291 digits x 256 Pixels SNMF has clustering like-properties (As reported in Ding, 2005) SNMF have part based representation (As reported in Lee&Seung, 1999) EUSIPCO’09 27 August 2009

14. Conclusion • Bayesian learning forms a simple framework for tuning the pruning in sparse NMF thereby both establishing the model order as well as resolving the non-uniqueness of the NMF representation. • Likelihood function (i.e. KL (Poisson noise) vs. LS (Gaussian noise)) heavily impacted the extracted number of components.In comparison, a tensor decomposition study given in (Mørup et al., Journal of Chemometrics 2009) demonstrated that the choice of prior distribution only has limited effect for the model order estimation. • Many other conceivable parameterizations of the prior as well as approaches to parameter estimation. However, Bayesian learning forms a promising framework for model order estimation as well as resolving ambiguities in the NMF model through the tuning of the pruning. EUSIPCO’09 27 August 2009