1 / 50

Review of Linear Algebra

Review of Linear Algebra. Fall 2014 The University of Iowa Tianbao Yang. Announcements. TA: Shiyao Wang Office hours: 3:30- 5:00 pm Tu / Th Office Location: 201C Homework-1 is available on ICON. Today’s Topics. Vector and Matrix Operation on Matrices/Vectors

Télécharger la présentation

Review of Linear Algebra

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Review of Linear Algebra Fall 2014 The University of Iowa Tianbao Yang

  2. Announcements • TA: Shiyao Wang • Office hours: 3:30-5:00 pm Tu/Th • Office Location: 201C • Homework-1 is available on ICON

  3. Today’s Topics • Vectorand Matrix • Operation on Matrices/Vectors • Singular value decomposition • Norms • An Application in Text Analysis

  4. Vector • Scalar • a real number: 7 • Vector • one dimensional array • representation: column vector • representation: row vector

  5. Vector • Dimensionality or size: • number of scalars • Vector Space • all vectors of the same dimension

  6. Matrix • Two dimensional array • Representation • (i,j)-th element: • A set of vectors • vector: a special matrix rows columns

  7. Matrix • Dimensionality or size • m*n (m rows and n columns) • Matrix Space:

  8. Today’s Topics • Vectorand Matrix • Operation on Matrices/Vectors • Singular value decomposition • Norms • An Application in Text Analysis

  9. Operations • Matrix addition: • two matrices of the same size • (i,j)-th element: • Scalar multiplication: • results in the same size • Matrix subtraction:

  10. Operations • Multiplication of a row vector and a column vector • Matrix Multiplication • ,

  11. Operations

  12. Operations • Transpose: • (i,j)-the element: • transpose of a column vector: row vector • Rules:

  13. Special Matrices • Square matrix: • Symmetric matrix: • Zero matrix • all elements are zeros • Identity Matrix: • each column (or row) standard basis • :

  14. Operations • (Square) Matrix Inverse • similar to inverse of a scalar: • inverse of a square matrix: • if there exists: Non-singular

  15. Operations • Trace of a square matrix: • definition • rules

  16. Today’s Topics • Vectorand Matrix • Operation on Matrices/Vectors • Singular value decomposition • Norms • An Application in Text Analysis

  17. mm mm mn mn V is nn V is nn Singular Value Decomposition • A matrix: • Singular Value Decomposition (SVD) • The columns of are left singular vectors • The columns of are right singular vectors • is a diagonal matrix with singular values (positive values)

  18. mm mn V is nn Singular Value Decomposition • Illustration of SVD dimensions and sparseness

  19. Singular Value Decomposition • Rank of a Matrix • organize singular values in descending order • the largest index that is non-zero

  20. Eigen-value Decomposition • Eigenvectors(for a square mm matrix S) • Example (right) eigenvector eigenvalue

  21. Eigen-value Decomposition

  22. Eigen-value Decomposition S = U *  * UT

  23. Eigen-value Decomposition S = U *  * UT

  24. Eigen-value Decomposition • This is generally true for symmetric square matrix • Columns of U are eigenvectors of S • Diagonal elements of  are eigenvalues of S S = U *  * UT

  25. mm mn V is nn nn nn nn Eigen-value Decomposition • A symmetric matrix: • Eigen-value Decomposition • The columns of are eigen-vectors • is a diagonal matrix with real eigen-values

  26. mm mn V is nn nn nn nn Positive (Semi-)Definite Matrix • A symmetric matrix: • Eigen-value Decomposition • The columns of are eigen-vectors • is a diagonal matrix with Positive eigen-values • is a diagonal matrix with Non-negative eigen-values

  27. Today’s Topics • Vectorand Matrix • Operation on Matrices/Vectors • Singular value decomposition • Norms • An Application in Text Analysis

  28. Inner Product • inner product between two vectors • Norm of a Vector: (Euclidean Norm, norm)

  29. Inequalities • Cauchy-Schwarz Inequality • Triangle Inequality

  30. p-Norm of a Vector • p-norm • p = 1 norm • p = 2 norm • p =  norm

  31. Norm of a Matrix • Inner Product between two matrices • Norm of a Matrix (Frobenius norm)

  32. Other Matrix Norms • Induced Norm (operator norm): • p=2, spectral norm: maximum singular value • p=1, maximum absolute column sum • p= , maximum absolute row sum

  33. Other Matrix Norms • Schatten Norm: • p=1, trace norm (or nuclear norm) • p=2, Frobenius norm • p= , Spectral norm

  34. Machine Learning Problems • Solve the following problems Loss norm

  35. Today’s Topics • Vectorand Matrix • Operation on Matrices/Vectors • Singular value decomposition • Norms • An Application in Search Engine

  36. Search Engine • A database of Webpages • A user-typed query • generate a list of relevant webpages • A ranking problem https://www.facebook.com/ contain query words (LSI) a lot of links to them (PageRank)

  37. Representation of documents • webpage is a document • document contains many terms (words) • To represent a document • collect all meaningful terms • count the occurrence of each term in a document

  38. Representation of documents • Term-Document Matrix

  39. Search Engine • Represent the query in the same way • e.g. query: “computer system” Query 0 0 1 0 1 0 0 0 0 0 0 0

  40. Search Engine • Retrieve Similar Documents • Query • Similarity • inner product • normalized inner product (cosine similarity) • Assume A is column normalized and q is normalized

  41. Concept Concept Rep. of Concepts in term space Rep. of concepts in document space Search Engine • Latent Semantic Indexing • SVD

  42. Search Engine • Low rank approximation: • approximate matrix with the largest singular values and singular vectors Rank-k approximation

  43. Search Engine • Why Low rank approximation: • data compression: billions to thousands • filter out noise Rank-k approximation

  44. Finding “Good Concepts”

  45. X X SVD: Example: m=2

  46. X X LSI: Example: m=2

  47. X X LSI: Example: m=2

  48. X X LSI: Example: m=2

  49. LSI: Example: m=3 Top three left singular vectors -0.2214 -0.1132 0.2890 -0.1976 -0.0721 0.1350 -0.2405 0.0432 -0.1644 -0.4036 0.0571 -0.3378 -0.6445 -0.1673 0.3611 -0.2650 0.1072 -0.4260 -0.2650 0.1072 -0.4260 -0.3008 -0.1413 0.3303 -0.2059 0.2736 -0.1776 -0.0127 0.4902 0.2311 -0.0361 0.6228 0.2231 -0.0318 0.4505 0.1411

  50. Search Engine • Why Low rank approximation: • data compression: billions to thousands • filter out noise

More Related