Out-of-Sample Extension and Reconstruction on Manifolds

1. Out-of-Sample Extension and Reconstruction on Manifolds BhuwanDhingraFinal Year (Dual Degree)Dept of Electrical Engg.

2. Introduction • An m-dimensional manifold is a topological space which is locally homeomorphic to the m-dimensional Euclidean space • In this work we consider manifolds which are: • Differentiable • Embedded in a Euclidean space • Generated from a set of m latent variables via a smooth function f

3. Introduction n >> m

4. Non-Linear Dimensionality Reduction • In practice we only have a sampling on the manifold • Y is estimated using a Non-Linear Dimensionality Reduction (NLDR) method • Examples of NLDR methods –ISOMAP, LLE, KPCA etc. • However most non-linear methods only provide the embedding Y and not the mappings f and g

5. Problem Statement g y* x* f

6. Outline • p is the nearest neighbor of x* • Only the points in are used for extension and reconstruction

7. Outline • The tangent plane is estimated from the k-nearest neighbors of p using PCA

8. Out-of-Sample Extension • A linear transformation Aeis learnt s.t Y = AeZ • Embedding for new point y* = Aez* z* y* Ae

9. Out-of-Sample Reconstruction z* y* Ar • A linear transformation Aris learnt s.t Z = ArY • Projection of reconstruction on tangent plane z* = Ary*

10. Principal Components Analysis • Covariance matrix of neighborhood: • Let be the eigenvector and eigenvalue matrixes of Mk • Then • Denote then the projection of a point x onto the tangent plane is given by:

11. Linear Transformation • Y and Z are both centered around and • Then Ae =BeRewhere Be and Re are scale and rotation matrices respectively • If is the singular value decomposition of ZTY, then

12. Final Estimates

13. Error Analysis • We don’t know the true form of f or g to compare our estimates against • Reconstruction Error: For a new point x* its reconstruction is computed as , and the reconstruction error is

14. Sampling Density • To show: As the sampling density of points on the manifold increases, reconstruction error of a new point goes to 0 • In a k-NN framework, the sampling density can increase in two ways: • k remains fixed and the sampling width decereases • remains fixed and • We consider the second case

15. Neighborhood Parameterization • Assume that the first m-canonical vectors of are along

16. Reconstruction Error • But ArAe = I, hence

17. Reconstruction Error • Tyagi, Vural and Frossard (2012) derive conditions on k s.t the angle between and is bounded • They show that as • Equivalently, where Rm is an aribitrarym-dimensional rotation matrix • and

18. Reconstruction Error • Hence the reconstruction approaches the projection of x* onto

19. Smoothness of Manifold • If the manifold is smooth then all will be smooth • Taylor series of : • As because x* will move closer to p

20. Results - Extension • Out of sample extension on the Swiss-Roll dataset • Neighborhood size = 10

21. Results - Extension • Out of sample extension on the Japanese flag dataset • Neighborhood size = 10

22. Results - Reconstruction • Reconstructions of ISOMAP faces dataset (698 images) • n = 4096, m = 3 • Neighborhood size = 8

23. Reconstruction error v Number of Points on Manifold • ISOMAP Faces dataset • Number of cross validation sets = 5 • Neighborhood size = [6, 7, 8, 9]