Understanding PCA, pPCA, and ICA in Dimensionality Reduction and Signal Separation
This lecture covers Principal Component Analysis (PCA), Probabilistic PCA (pPCA), and Independent Component Analysis (ICA)—essential techniques in data analysis. PCA helps identify the subspace that maximizes data variance, facilitating dimensionality reduction through eigenvalue decomposition of the sample covariance matrix. pPCA extends PCA with a probabilistic framework. ICA focuses on separating non-Gaussian signals, such as audio mixtures, highlighting the importance of independent source distributions. Key methods like gradient descent and the natural gradient are discussed for efficient learning.
Understanding PCA, pPCA, and ICA in Dimensionality Reduction and Signal Separation
E N D
Presentation Transcript
Lecture 14 PCA, pPCA, ICA
Principal Components Analysis • PCA is a data analysis technique to find the subspace of input space that carries most of the variance of the data. • It is therefore useful as a tool to reduce the dimensionality of input space. • The solution is found by an eigen-value decomposition of the sample covariance matrix. • PPCA is a probabilistic model that has ML solution equal to the PCA solution. It is a special case of FA with isotropic variance. • Therefore, the EM algorithm for FA is applicable for learning.
Independent Component Analysis • FA, PPCA have Gaussian prior models. In ICA we use non-Gaussian prior models (i.e. heavy tailed or bi-modal). • We also do not insist on dimensionality reduction, although that is also possible, but not necessary. • The canonical example is 2 speakers producing different mixtures of sound in 2 microphones that we wish to unmix. • The source distributions are non-Gaussian but independent, the noise model is typically Gaussian. • The simplest ICA model is square and has no noise. We can use a change of variable to go from sources to inputs. • Learning is through gradient descend with the ``natural gradient’’.
