INFORMATION RETRIEVAL TECHNIQUES BY DR. ADNAN ABID

1. INFORMATION RETRIEVAL TECHNIQUESBYDR. ADNAN ABID Lecture # 4 Vector Space Retrieval Model

2. ACKNOWLEDGEMENTS The presentation of this lecture has been taken from the following sources • “Introduction to information retrieval” by PrabhakarRaghavan, Christopher D. Manning, and Hinrich Schütze • “Managing gigabytes” by Ian H. Witten, ‎Alistair Moffat, ‎Timothy C. Bell • “Modern information retrieval” by Baeza-Yates Ricardo, ‎ • “Web Information Retrieval” by Stefano Ceri, ‎Alessandro Bozzon, ‎Marco Brambilla

3. Outline • The Vector Model • Term frequency tf • Document frequency • tf-idf weighting

4. The Vector Model (Salton, 1968) • The Vector Space Model represents documents and queries as vectors in the term space • Term weights are used to compute the degree of similarity (or score) between each document stored in the system and the user query • By sorting in decreasing order of this degree of similarity, the vector model takes into consideration documents which match the query terms only partially. • Ranked document answer set is a lot more precise (in the sense that it better matches the user information need) than the document answer set retrieved by the Boolean model • A measure of the similarity between the two vectors is then computed • Documents whose content (terms in the document) correspond most closely to the content of the query are judged to be the most relevant

5. Binary term-document incidence matrix Each document is represented by a binary vector ∈ {0,1}|V|

6. Term-document count matrices • Consider the number of occurrences of a term in a document: • Each document is a count vector in ℕv: a column below

7. Bag of words model • Vector representation doesn’t consider the ordering of words in a document • John is quicker than Maryand Mary is quicker than John have the same vectors • This is called the bag of words model. • In a sense, this is a step back: The positional index is required to distinguish these two documents. • We will look at “recovering” positional information later in this course. • For now: bag of words model

8. Term frequency tf • The term frequency tft,d of term t in document d is defined as the number of times that t occurs in d. • We want to use tf when computing query-document match scores. But how? • Raw term frequency is not what we want: • A document with 10 occurrences of the term is more relevant than a document with 1 occurrence of the term. • But not 10 times more relevant. • Relevance does not increase proportionally with term frequency. NB: frequency = count in IR

9. Log-frequency weighting • The log frequency weight of term t in d is • 0 → 0, 1 → 1, 2 → 1.3, 10 → 2, 1000 → 4, etc.

10. Term Frequency Normalization • We can use another normalization method for tf. • Find the term frequencies of all terms in a document. • Find out the maximum tfmax from all those terms. • The tf of term ‘i’ referred to as tfi is obtained by dividing the tfi by tfmax. tfi= tfi/tfmax • This brings the term frequencies of each term in a document between 1 and 0, irrespective of the size of a document.

11. Document frequency • Rare terms are more informative than frequent terms • Stop words • Consider a term in the query that is rare in the collection (e.g., arachnocentric) • A document containing this term is very likely to be relevant to the query arachnocentric • → We want a high weight for rare terms like arachnocentric.

12. Document frequency, continued • Frequent terms are less informative than rare terms • Consider a query term that is frequent in the collection (e.g., high, increase, line) • A document containing such a term is more likely to be relevant than a document that doesn’t • But it’s not a sure indicator of relevance. • → For frequent terms, we want high positive weights for words like high, increase, and line • But lower weights than for rare terms. • We will use document frequency (df) to capture this.

13. idf weight • dft is the document frequency of t: the number of documents that contain t • dft is an inverse measure of the informativeness of t • dft N • We define the idf (inverse document frequency) of t by • We use log (N/dft) instead of N/dft to “dampen” the effect of idf. Will turn out the base of the log is immaterial.

14. idf example, suppose N = 1 million There is one idf value for each term t in a collection.

15. tf-idf weighting • The tf-idf weight of a term is the product of its tf weight and its idf weight. • Best known weighting scheme in information retrieval • Note: the “-” in tf-idf is a hyphen, not a minus sign! • Alternative names: tf.idf, tf x idf • Increases with the number of occurrences within a document • Increases with the rarity of the term in the collection

16. Resources • IIR 6.2 – 6.4.3 • http://www.miislita.com/information-retrieval-tutorial/cosine-similarity-tutorial.html • Term weighting and cosine similarity tutorial for SEO folk!