Optimality of Singular Value Decomposition (SVD) in Matrix Approximation

1. Multimedia Databases SVD II

2. Optimality of SVD Def: TheFrobenius norm of a n x m matrix M is (reminder) The rank of a matrix M is the number of independent rows (or columns) of M Let A=ULVT and Ak = UkLk VkT (SVD approximation of A) Ak is annxm matrix, Uk an nxk, Lk kxk, and Vk mxk Theorem: [Eckart and Young] Among all n x m matrices C of rank at most k, we have that:

3. SVD - Other properties - summary • can produce orthogonal basis • can solve over- and under-determined linear problems (see C(1) property) • can compute ‘fixed points’ (= ‘steady state prob. in Markov chains’) (see C(4) property)

4. SVD -outline of properties • (A): obvious • (B): less obvious • (C): least obvious (and most powerful!)

5. Properties - by defn.: A(0): A[n x m] = U[ n x r ]L[ r x r ]VT[ r x m] A(1): UT [r x n] U [n x r ] = I [r x r ] (identity matrix) A(2): VT [r x n] V [n x r ] = I [r x r ] A(3): Lk = diag( l1k, l2k, ... lrk ) (k: ANY real number) A(4): AT= V L UT

6. Less obvious properties A(0): A[n x m] = U[ n x r ]L[ r x r ]VT[ r x m] B(1): A[n x m] (AT) [m x n] = ??

7. Less obvious properties A(0): A[n x m] = U[ n x r ]L[ r x r ]VT[ r x m] B(1): A[n x m] (AT) [m x n] = UL2UT symmetric; Intuition?

8. Less obvious properties A(0): A[n x m] = U[ n x r ]L[ r x r ]VT[ r x m] B(1): A[n x m] (AT) [m x n] = UL2UT symmetric; Intuition? ‘document-to-document’ similarity matrix B(2): symmetrically, for ‘V’ (AT) [m x n] A[n x m] = VL2VT Intuition?

9. Less obvious properties B(3): ((AT) [m x n] A[n x m] )k= VL2kVT and B(4): (ATA )k ~ v1l12kv1T for k>>1 where v1: [m x 1] first column (eigenvector) of V l1: strongest eigenvalue

10. Less obvious properties B(4): (ATA )k ~ v1l12kv1T for k>>1 B(5): (ATA )k v’ ~ (constant) v1 ie., for (almost) any v’, it converges to a vector parallel to v1 Thus, useful to compute first eigenvector/value (as well as the next ones…)

11. Less obvious properties - repeated: A(0): A[n x m] = U[ n x r ]L[ r x r ]VT[ r x m] B(1): A[n x m] (AT) [m x n] = UL2UT B(2): (AT) [m x n] A[n x m] = VL2VT B(3): ((AT) [m x n] A[n x m] )k= VL2kVT B(4): (ATA )k ~ v1l12k v1T B(5): (ATA )k v’ ~ (constant) v1

12. Least obvious properties A(0): A[n x m] = U[ n x r ]L[ r x r ]VT[ r x m] C(1): A[n x m]x [m x 1] = b [n x 1] let x0 = VL(-1)UTb if under-specified, x0 gives ‘shortest’ solution if over-specified, it gives the ‘solution’ with the smallest least squares error

13. Least obvious properties Illustration: under-specified, eg [1 2] [w z] T = 4 (ie, 1 w + 2 z = 4) z shortest-length solution x0 all possible solutions 2 1 1 2 3 4 w

14. Least obvious properties - cont’d A(0): A[n x m] = U[ n x r ]L[ r x r ]VT[ r x m] C(2): A[n x m]v1 [m x 1] = l1 u1[n x 1] where v1 , u1 the first (column) vectors of V, U. (v1== right-eigenvector) C(3): symmetrically: u1T A = l1 v1T u1 == left-eigenvector Therefore:

15. Least obvious properties - cont’d A(0): A[n x m] = U[ n x r ]L[ r x r ]VT[ r x m] C(4): AT Av1= l12 v1 (fixed point - the usual dfn of eigenvector for a symmetric matrix)

16. Least obvious properties - altogether A(0): A[n x m] = U[ n x r ]L[ r x r ]VT[ r x m] C(1): A[n x m]x [m x 1] = b [n x 1] then, x0 = VL(-1)UTb: shortest, actual or least-squares solution C(2): A[n x m]v1 [m x 1] = l1 u1[n x 1] C(3): u1T A = l1 v1T C(4): AT Av1= l12 v1

17. Kleinberg’s Algorithm • Main idea: In many cases, when you search the web using some terms, the most relevant pages may not contain this term (or contain the term only a few times) • Harvard : www.harvard.edu • Search Engines: yahoo, google, altavista • Authorities and hubs

18. Kleinberg’s algorithm • Problem dfn: given the web and a query • find the most ‘authoritative’ web pages for this query Step 0: find all pages containing the query terms (root set) Step 1: expand by one move forward and backward (base set)

19. Kleinberg’s algorithm • Step 1: expand by one move forward and backward

20. Kleinberg’s algorithm • on the resulting graph, give high score (= ‘authorities’) to nodes that many important nodes point to • give high importance score (‘hubs’) to nodes that point to good ‘authorities’) hubs authorities

21. Kleinberg’s algorithm observations • recursive definition! • each node (say, ‘i’-th node) has both an authoritativeness score ai and a hubness score hi

22. Kleinberg’s algorithm Let E be the set of edges and A be the adjacency matrix: the (i,j) is 1 if the edge from i to j exists Let h and a be [n x 1] vectors with the ‘hubness’ and ‘authoritativiness’ scores. Then:

23. Kleinberg’s algorithm Then: ai = hk + hl + hm that is ai = Sum (hj) over all j that (j,i) edge exists or a = ATh k i l m

24. Kleinberg’s algorithm symmetrically, for the ‘hubness’: hi = an + ap + aq that is hi = Sum (qj) over all j that (i,j) edge exists or h = Aa i n p q

25. Kleinberg’s algorithm In conclusion, we want vectors h and a such that: h = Aa a = ATh Recall properties: C(2): A[n x m]v1 [m x 1] = l1 u1[n x 1] C(3): u1T A = l1 v1T

26. Kleinberg’s algorithm In short, the solutions to h = Aa a = ATh are the left- and right- eigenvectors of the adjacency matrix A. Starting from random a’ and iterating, we’ll eventually converge (Q: to which of all the eigenvectors? why?)

27. Kleinberg’s algorithm (Q: to which of all the eigenvectors? why?) A: to the ones of the strongest eigenvalue, because of property B(5): B(5): (ATA )k v’ ~ (constant) v1

28. Kleinberg’s algorithm - results Eg., for the query ‘java’: 0.328 www.gamelan.com 0.251 java.sun.com 0.190 www.digitalfocus.com (“the java developer”)

29. Kleinberg’s algorithm - discussion • ‘authority’ score can be used to find ‘similar pages’ to page p • closely related to ‘citation analysis’, social networs / ‘small world’ phenomena

30. google/page-rank algorithm • closely related: The Web is a directed graph of connected nodes • imagine a particle randomly moving along the edges (*) • compute its steady-state probabilities. That gives the PageRank of each pages (the importance of this page) (*) with occasional random jumps

31. PageRank Definition • Assume a page A and pages T1, T2, …, Tm that point to A. Let d is a damping factor. PR(A) the pagerank of A. C(A) the out-degree of A. Then:

32. google/page-rank algorithm • Compute the PR of each page~identical problem: given a Markov Chain, compute the steady state probabilities p1 ... p5 2 1 3 4 5

33. Computing PageRank • Iterative procedure • Also, … navigate the web by randomly follow links or with prob p jump to a random page. Let A the adjacency matrix (n x n), di out-degree of page i Prob(Ai->Aj) = pn-1+(1-p)di–1Aij A’[i,j] = Prob(Ai->Aj)

34. google/page-rank algorithm • Let A’ be the transition matrix (= adjacency matrix, row-normalized : sum of each row = 1) 2 1 3 = 4 5

35. google/page-rank algorithm • A p = p A p = p 2 1 3 = 4 5

36. google/page-rank algorithm • A p = p • thus, p is the eigenvector that corresponds to the highest eigenvalue (=1, since the matrix is row-normalized)

37. Kleinberg/google - conclusions SVD helps in graph analysis: hub/authority scores: strongest left- and right- eigenvectors of the adjacency matrix random walk on a graph: steady state probabilities are given by the strongest eigenvector of the transition matrix

38. Conclusions • SVD: a valuable tool • given a document-term matrix, it finds ‘concepts’ (LSI) • ... and can reduce dimensionality (KL) • ... and can find rules (PCA; RatioRules)

39. Conclusions cont’d • ... and can find fixed-points or steady-state probabilities (google/ Kleinberg/ Markov Chains) • ... and can solve optimally over- and under-constraint linear systems (least squares)

40. Brin, S. and L. Page (1998). Anatomy of a Large-Scale Hypertextual Web Search Engine. 7th Intl World Wide Web Conf. References