SlideServe
Télécharger Obtenir Pro
Menu
Accueil Pro
Préférences
ApparenceLight Aide et commentaires
© SlideServe · v2026

Statistical Properties for Text

kacy avatar
kacy
20 diapositives · 371 Views · Jul 24, 2014

This article explores the statistical properties of text, focusing on the distribution of word frequencies and the growth of vocabulary size within a corpus. Key concepts, such as Zipf's Law and Heaps' Law, are discussed to highlight how certain words dominate occurrences and how vocabulary expands in relation to corpus size. The implications of these properties for information retrieval systems are examined, including term weighting and handling rare words. Real-world data comparisons illustrate the effectiveness of Zipf's Law and its impact on linguistic analysis and information storage efficiency.

Statistical Properties for Text - slide 1
01 / 20Statistical Properties for Text
1 / 17

Télécharger la présentation

Statistical Properties for Text
Une image / un lien ci-dessous est fourni (tel quel) pour télécharger la présentation

Politique de téléchargement : le contenu du site vous est fourni TEL QUEL pour votre information et votre usage personnel ; il ne peut être vendu, concédé sous licence ni partagé sur d’autres sites sans l’accord de son auteur.Le contenu vous est fourni TEL QUEL, pour votre information et votre usage personnel uniquement. Téléchargez la présentation en cliquant sur ce lien. Lors du téléchargement, si vous ne parvenez pas à télécharger une présentation, l’éditeur a peut-être supprimé le fichier de son serveur.Pendant le téléchargement, si vous ne parvenez pas à obtenir une présentation, le fichier a peut-être été supprimé par l’éditeur.

F I N

Plus de contenu associé

Transcription de la présentation

  1. Statistical Properties for Text Rong Jin

  2. Statistical Properties of Text • How is the frequency of different words distributed? • How fast does vocabulary size grow with the size of a corpus? • Such factors affect the performance of information retrieval and can be used to select appropriate term weights and other aspects of an IR system.

  3. Sample Word Frequency Data(from B. Croft, UMass)

  4. Zipf’s Law • A few words occur very often • 2 most frequent words can account for 10% of occurrences • top 6 words are 20%, top 50 words are 50% • Many words are infrequent • “Principle of Least Effort” • easier to repeat words rather than coining new ones

  5. Zipf’s Law Collected from collection WSJ 1987

  6. Zipf’s Law • Rank · Frequency ≈ Constant pr = (Number of occurrences of word of rank r)/N • N total word occurrences • probability that a word chosen randomly from the text will be the word of rank r r ·pr = A • A ≈ 0.1

  7. Zipf and Term Weighting • Both extremely common and extremely uncommon words were not very useful for indexing.

  8. Predicting Occurrence Frequencies • By Zipf, a word appearing n times has rank rn=AN/n • Several words may occur n times, assume rank rn applies to the last of these. • Therefore, rn words occur n or more times and rn+1 words occur n+1 or more times. • So, the number of words appearing exactlyn times is:

  9. Predicting Word Frequencies (cont) • Assume highest ranking term occurs once and therefore has rank D = AN/1 • Fraction of words with frequency n is: • Fraction of words appearing only once is therefore ½.

  10. Occurrence Frequency Data(from B. Croft, UMass)

  11. Does Real Data Fit Zipf’s Law? • A law of the form y = kxc is called a power law. • Zipf’s law is a power law with c = –1 • On a log-log plot, power laws give a straight line with slope c. • Zipf is quite accurate except for very high and low rank.

  12. Fit to Zipf for Brown Corpus • How to fit a straight line in log-log plot ?

  13. Fit to Zipf for Brown Corpus

  14. Mandelbrot Fit

  15. Explanations for Zipf’s Law • Zipf’s explanation was his “principle of least effort.” Balance between speaker’s desire for a small vocabulary and hearer’s desire for a large one. • Simon explanation is “rich get richer.” • Li (1992) shows that just random typing of letters including a space will generate “words” with a Zipfian distribution. • http://linkage.rockefeller.edu/wli/zipf/

  16. Zipf’s Law Impact on IR • Good News: • Stopwords will account for a large fraction of text so eliminating them greatly reduces inverted-index storage costs. • Bad News: • For most words, gathering sufficient data for meaningful statistical analysis (e.g. for correlation analysis for query expansion) is difficult since they are extremely rare.

  17. Prevalence of Zipfian Laws • Many items exhibit a Zipfian distribution. • Population of cities • Wealth of individuals • Discovered by sociologist/economist Pareto in 1909 • Popularity of books, movies, music, web-pages, etc. • Popularity of consumer products • Chris Anderson’s “long tail”

  18. Vocabulary Growth • How does the size of the overall vocabulary (number of unique words) grow with the size of the corpus? • This determines how the index size will scale with the size of the corpus. • Vocabulary not really upper-bounded due to proper names, typos, etc.

  19. Heaps’ Law

  20. Heaps’ Law • If V is the size of the vocabulary and the n is the length of the corpus in words: • Typical constants: • K 10100 •   0.40.6 (approx. square-root)