1 / 23

Understanding Language Models in Information Retrieval

This document presents an overview of language models used in information retrieval, focusing on their types, such as Unigram and Bigram models, and the Query Likelihood Model. It discusses the importance of smoothing techniques to handle unseen words and evaluates the models using precision and recall metrics. Various models are compared, highlighting their mathematical and conceptual advantages while recognizing limitations in relevancy capture. Examples illustrate the application of these models in ranking documents based on the probability of queries.

zorita-craig
Télécharger la présentation

Understanding Language Models in Information Retrieval

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Language Models for Information Retrieval Andy Luong and Nikita Sudan

  2. Outline • Language Model • Types of Language Models • Query Likelihood Model • Smoothing • Evaluation • Comparison with other approaches

  3. Language Model • A language model is a function that puts a probability measure over strings drawn from some vocabulary.

  4. Language Models P(q|Md) instead of P(R=1|q,d)

  5. Example • Doc1: “frog said that toad likes frog” • Doc2: “toad likes frog” 1/3 1/6 1/3

  6. Example Continued q = “frog likes toad” P(q | M1) = (1/3)*(1/6)*(1/6)*0.8*0.8*0.2 P(q | M2) = (1/3)*(1/3)*(1/3)*0.8*0.8*0.2 P(q | M1) < P (S | M2)

  7. Types of Language Models CHAIN RULE UNIGRAM LM BIGRAM LM

  8. Multinomial distribution Frequency Order Constraint M is the size of the term vocabulary

  9. Query Likelihood Model ≈

  10. Query Likelihood Model • Infer LM for each document • Estimate P(q | Md(i)) • Rank documents based on probabilities

  11. MLE

  12. Smoothing • Basic Intuition • New word or unseen word in the document • P( t | Md) = 0 • Zero probabilities will make P ( q | Md) = 0 • Why else should we smooth?

  13. Smoothing Continued Non-occurring term Probability Bound Linear Interpolation Language Model

  14. Example • Doc1: “frog said that toad likes frog” • Doc2: “toad likes frog” 1/3 1/9 1/9 2/9 2/9

  15. Example Continued q= “frog said” λ = ½ P(q | M1) = [(1/3 + 1/3)*(1/2)] * [(1/6 + 1/9)*(1/2)] = .046 P(q | M2) = [(1/3 + 1/3)*(1/2)] * [(0 + 1/9)*(1/2)] = .018 P(q | M1) > P (q | M2)

  16. Evaluation • Precision = (relevant documents ∩ retrieved documents)/ retrieved documents • Recall = (relevant documents ∩ retrieved documents)/ relevant documents

  17. Tf-Idf • The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.

  18. Ponte and Croft’s Experiments

  19. Pros and Cons • “Mathematically precise, conceptually simple, computationally tractable and intuitively appealing.” • Relevancy is not captured

  20. Query vs. Document Model (a) Query Likelihood (b) Document Likelihood (c) Model Comparison

  21. KL divergence

  22. Thank you.

  23. Questions?

More Related