Dimensionality reduction

1. Dimensionality reduction CISC 5800 Professor Daniel Leeds

2. The benefits of extra dimensions • Finds existing complex separations between classes

3. The risks of too-many dimensions • High dimensions with kernels over-fit the outlier data • Two dimensions ignore the outlier data

4. Training vs. testing • Training: learn parameters from set of data in each class • Testing: measure how often classifier correctly identifies new data • More training reduces classifier error • More gradient ascent steps • More learned feature • Too much training causes worse testing error – overfitting error training epochs

5. Goal: High Performance, Few Parameters • “Information criterion”: performance/parameter trade-off • Variables to consider: • L likelihood of train data after learning • k number of parameters (e.g., number of features) • m number of points of training data • Popular information criteria: • Akaike information criterion AIC: log(L) - k • Bayesian information criterion BIC: log(L) - 0.5 k log(m)

6. Decreasing parameters • Force parameter values to 0 • L1 regularization • Support Vector selection • Feature selection/removal • Consolidate feature space • Component analysis

7. Feature removal • Start with feature set: F={x1, …, xk} • Find classifier performance with set F: perform(F) • Loop • Find classifier performance for removing feature x1, x2, …, xk: argmaxi perform(F-x1) • Remove feature that causes least decrease in performance:F=F-xi Repeat, using AIC or BIC as termination criterion • AIC: log(L) - k • BIC: log(L) - 0.5 k log(m)

8. AIC testing: log(L)-k

9. Feature selection • Find classifier performance for just set of 1 feature: argmaxi perform({xi}) • Add feature with highest performance: F={xi} • Loop • Find classifier performance for adding one new feature: argmaxi perform(F+{xi}) • Add to F feature with highest performance increase: F=F+{xi} Repeat, using AIC or BIC as termination criterion • AIC: log(L) - k • BIC: log(L) - 0.5 k log(m)

10. Defining new feature axes • Identify a common trend • Map data onto new dimension u1

11. Defining data points with new axes

12. Component analysis Each data point xi in D can be reconstructed as sum of components u: • is weight on qth component to reconstruct data point xi

13. Component analysis: examples Data Components 2 -1 0

14. Component analysis: examples “Eigenfaces” – learned from set of face images u: nine components xi: data reconstructed

15. Types of component analysis Learn new axes from data sets: common “components” • Principal component analysis (PCA): • Best reconstruction of each data point xiwith first t components • Each component perpendicular to all others: • Independent component analysis (ICA): • Minimize number of components to describe each • Can focus on different components for different • Non-negative matrix factorization (NMF): • All data xi non-negative • All components and weights non-negative

16. Principle component analysis (PCA) Start with • D = {x1,…,xn} , data 0-center • Component index: q=1 Loop • Find direction of highest variance: • Ensure • Remove uqfrom data: We require Thus, we guarantee

17. Independent component analysis (ICA) Start with • D = {x1,…,xn} , data 0-center Find group(s) for each data point Find direction for each group uq • Ensure We do not require Thus, we cannot guarantee

18. Evaluating components • Components learned in order of descriptive power • Compute reconstruction error for all data by using first v components: