1 / 61

Probabilistic Models for Text Analysis: Overview

Explore various probabilistic models for text analysis, from latent semantic analysis to Latent Dirichlet Allocation, and applications in hyperlink modeling. Learn about tools, techniques, and challenges in modeling text data effectively.

nicki
Télécharger la présentation

Probabilistic Models for Text Analysis: Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Modeling Text and Links: Overview William W. Cohen Machine Learning Dept. and Language Technologies Institute School of Computer Science Carnegie Mellon University Joint work with: Amr Ahmed, Andrew Arnold, Ramnath Balasubramanyan, Frank Lin, Matt Hurst (MSFT), Ramesh Nallapati, Noah Smith, Eric Xing, Tae Yano

  2. Outline • Tools for analysis of text • Probabilistic models for text, communities, and time • Mixture models and LDA models for text • LDA extensions to model hyperlink structure

  3. Introduction to Topic Models

  4. Introduction to Topic Models • Probabilistic Latent Semantic Analysis (PLSA) d d • Select document d ~ Mult() • For each position n = 1,, Nd • generate zn ~ Mult( z| d) • generate wn ~ Mult( z| zn)  Doc-specific z Topic distribution • Mixture model: • each document is generated by a single (unknown) multinomial distribution of words, the corpus is “mixed” by  • PLSA model: • each word is generated by a single unknown multinomial distribution of words, each document is mixed by d w N M 

  5. Introduction to Topic Models • PLSA topics (TDT-1 corpus)

  6. Introduction to Topic Models JMLR, 2003

  7. Introduction to Topic Models • Latent Dirichlet Allocation  • For each document d = 1,,M • Generate d ~ Dir(¢ | ) • For each position n = 1,, Nd • generate zn ~ Mult( z| d) • generate wn ~ Mult( w| zn) a z w N M 

  8. Introduction to Topic Models • Perplexity comparison of various models Unigram Mixture model PLSA Lower is better LDA

  9. Introduction to Topic Models • Prediction accuracy for classification using learning with topic-models as features Higher is better

  10. Outline • Tools for analysis of text • Probabilistic models for text, communities, and time • Mixture models and LDA models for text • LDA extensions to model hyperlink structure • LDA extensions to model time • Alternative framework based on graph analysis to model time & community • Preliminary results & tradeoffs • Discussion of results & challenges

  11. Hyperlink modeling using PLSA

  12. Hyperlink modeling using PLSA[Cohn and Hoffman, NIPS, 2001]  • Select document d ~ Mult() • For each position n = 1,, Nd • generate zn ~ Mult( ¢ | d) • generate wn ~ Mult( ¢ | zn) • For each citation j = 1,, Ld • generate zj ~ Mult( ¢ | d) • generate cj ~ Mult( ¢ | zj) d d z z w c N L M  g

  13. Hyperlink modeling using PLSA[Cohn and Hoffman, NIPS, 2001]  PLSA likelihood: d d z z New likelihood: w c N L M  g Learning using EM

  14. Hyperlink modeling using PLSA[Cohn and Hoffman, NIPS, 2001] Heuristic:  (1-) 0 ·· 1 determines the relative importance of content and hyperlinks

  15. Hyperlink modeling using PLSA[Cohn and Hoffman, NIPS, 2001] • Experiments: Text Classification • Datasets: • Web KB • 6000 CS dept web pages with hyperlinks • 6 Classes: faculty, course, student, staff, etc. • Cora • 2000 Machine learning abstracts with citations • 7 classes: sub-areas of machine learning • Methodology: • Learn the model on complete data and obtain d for each document • Test documents classified into the label of the nearest neighbor in training set • Distance measured as cosine similarity in the  space • Measure the performance as a function of 

  16. Hyperlink modeling using PLSA[Cohn and Hoffman, NIPS, 2001] • Classification performance content Hyperlink Hyperlink content

  17. Hyperlink modeling using LDA

  18. Hyperlink modeling using LinkLDA[Erosheva, Fienberg, Lafferty, PNAS, 2004] a  • For each document d = 1,,M • Generate d ~ Dir(¢ | ) • For each position n = 1,, Nd • generate zn ~ Mult( ¢ | d) • generate wn ~ Mult( ¢ | zn) • For each citation j = 1,, Ld • generate zj ~ Mult( . | d) • generate cj ~ Mult( . | zj) z z w c N L M  g Learning using variational EM

  19. Hyperlink modeling using LDA[Erosheva, Fienberg, Lafferty, PNAS, 2004]

  20. Goals of analysis: Extract information about events from text “Understanding” text requires understanding “typical reader” conventions for communicating with him/her Prior knowledge, background, … Goals of analysis: Very diverse Evaluation is difficult And requires revisiting often as goals evolve Often “understanding” social text requires understanding a community NewswireText Social MediaText Science as a testbed for social text: an open community which we understand

  21. Author-Topic Model for Scientific Literature

  22. Author-Topic Model for Scientific Literature[Rozen-Zvi, Griffiths, Steyvers, Smyth UAI, 2004] a P • For each author a = 1,,A • Generate a ~ Dir(¢ | ) • For each topic k = 1,,K • Generate fk ~ Dir( ¢ | ) • For each document d = 1,,M • For each position n = 1,, Nd • Generate author x ~ Unif(¢ | ad) • generate zn ~ Mult( ¢ | a) • generate wn ~ Mult( ¢ | fzn) a x z  A w N M f b

  23. Author-Topic Model for Scientific Literature[Rozen-Zvi, Griffiths, Steyvers, Smyth UAI, 2004] a Learning: Gibbs sampling P  x z  A w N M f b

  24. Author-Topic Model for Scientific Literature[Rozen-Zvi, Griffiths, Steyvers, Smyth UAI, 2004] • Topic-Author visualization

  25. Stopped here

  26. Author-Topic Model for Scientific Literature[Rozen-Zvi, Griffiths, Steyvers, Smyth UAI, 2004] • Application 1: Author similarity

  27. Author-Topic-Recipient model for email data [McCallum, Corrada-Emmanuel,Wang, ICJAI’05]

  28. Author-Topic-Recipient model for email data [McCallum, Corrada-Emmanuel,Wang, ICJAI’05] Gibbs sampling

  29. Author-Topic-Recipient model for email data [McCallum, Corrada-Emmanuel,Wang, ICJAI’05] • Datasets • Enron email data • 23,488 messages between 147 users • McCallum’s personal email • 23,488(?) messages with 128 authors

  30. Author-Topic-Recipient model for email data [McCallum, Corrada-Emmanuel,Wang, ICJAI’05] • Topic Visualization: Enron set

  31. Author-Topic-Recipient model for email data [McCallum, Corrada-Emmanuel,Wang, ICJAI’05] • Topic Visualization: McCallum’s data

  32. Modeling Citation Influences

  33. Modeling Citation Influences[Dietz, Bickel, Scheffer, ICML 2007] • Copycat model of citation influence • LDA model for cited papers • Extended LDA model for citing papers • For each word, depending on coin flip c, you might chose to copy a word from a cited paper instead of generating the word

  34. Modeling Citation Influences[Dietz, Bickel, Scheffer, ICML 2007] • Citation influence model

  35. Modeling Citation Influences[Dietz, Bickel, Scheffer, ICML 2007] • Citation influence graph for LDA paper

  36. Amr Ahmed Eric Xing Models of hypertext for blogs [ICWSM 2008] Ramesh Nallapati me

  37. LinkLDA model for citing documents Variant of PLSA model for cited documents Topics are shared between citing, cited Links depend on topics in two documents Link-PLSA-LDA

  38. Experiments • 8.4M blog postings in Nielsen/Buzzmetrics corpus • Collected over three weeks summer 2005 • Selected all postings with >=2 inlinks or >=2 outlinks • 2248 citing (2+ outlinks), 1777 cited documents (2+ inlinks) • Only 68 in both sets, which are duplicated • Fit model using variational EM

  39. Topics in blogs Model can answer questions like: which blogs are most likely to be cited when discussing topic z?

  40. Topics in blogs Model can be evaluated by predicting which links an author will include in a an article Link-LDA Link-PLDA-LDA Lower is better

  41. a   z  z z z w c w N N  Another model: Pairwise Link-LDA • LDA for both cited and citing documents • Generate an indicator for every pair of docs • Vs. generating pairs of docs • Link depends on the mixing components (’s) • stochastic block model

  42. Pairwise Link-LDA supports new inferences… …but doesn’t perform better on link prediction

  43. Outline • Tools for analysis of text • Probabilistic models for text, communities, and time • Mixture models and LDA models for text • LDA extensions to model hyperlink structure • Observation: these models can be used for many purposes… • LDA extensions to model time • Alternative framework based on graph analysis to model time & community • Discussion of results & challenges

  44. Authors are using a number of clever tricks for inference….

  45. Predicting Response to Political Blog Posts with Topic Models [NAACL ’09] Noah Smith Tae Yano

  46. Political blogs and and comments Posts are often coupled with commentsections Comment style is casual, creative, less carefully edited 50

More Related