Basics of Information Retrieval - Focus: the Web

1. Basics of Information Retrieval - Focus: the Web Lillian N. Cassel February 2008 For CSC 2500 : Survey of Information Science A number of these slides are taken or adapted from Source: http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html

2. Basic ideas • Information overload • The challenging byproduct of the information age • Huge amounts of information available -- how to find what you need when you need it • What kinds of information do you manage? • What do you expect to find when you want/need it? • How frequently do you access each kind? • Think about addresses, e-mail messages, files of interesting articles, product brochures, business cards, photos, charts and tables, etc. • Information retrieval is the formal study of efficient and effective ways to extract the right bit of information from a collection. • The web is a special case, as we will discuss.

3. Some distinctions • Data, information, knowledge • How do you distinguish among them? • Data: symbols • Information: data that are processed to be useful; provides answers to "who", "what", "where", and "when" questions • Knowledge: application of data and information; answers "how" questions • Understanding: appreciation of "why” • Wisdom: evaluated understanding. • Russell Ackoff as reported inhttp://www.systems-thinking.org/dikw/dikw.htm • What are we seeking when we search the web?

4. Organization • Information sources - organization: • Very well organized, indexed, controlled • Give some examples • Totally unorganized, uncharacterized, uncontrolled • Give some examples • Something in between • Give some examples

5. Databases • Databases hold specific data items • Organization is explicit • Keys relate items to each other • Queries are constrained, but effective in retrieving the data that is there • Databases generally respond to specific queries with specific results • Browsing is difficult • Searching for items not anticipated by the designers can be difficult • Give an example of a database with which you interact regularly. • What is a query that works easily? • Have you tried unsuccessfully to get information you know is there?

6. The Web • The Web contains many kinds of elements • Organization? • There are no keys to relate items to each other • Queries are unconstrained; effectiveness depends on the tools used. • Web queries generally respond to general queries with specific results • Browsing is possible, though somewhat complicated • There are no designers of the overall Web structure. • Describe how you frequently use the web • What works easily? • What has been difficult?

7. How high is Mt Everest The answer, with source -- not just a link to the place where an answer could be found

8. How fast is the Web growing? An increase of 50 million sites during 2007 Source: http://news.netcraft.com/archives/web_server_survey.html

9. The servers What server software is providing content on the Web Source: http://news.netcraft.com/archives/web_server_survey.html

10. Digital Library • Something in between the very structured database and the unstructured Web. • Content is controlled. Someone makes the entries. (Maybe a lot of people make the entries, but there are rules for admission.) • Searching and browsing are somewhat open, not controlled by fixed keys and anticipated queries. • Nature of the collection regulates indexing somewhat.

11. Digital Library examples • American Memory: http://memory.loc.gov/ammem/index.html • The National Science Digital Library www.nsdl.org • CITIDEL: www.citidel.org

12. In all cases • Trying to connect an information user to the specific information wanted. • Concerned with efficiency and effectiveness • Effectiveness - how well did we do? • Efficiency - how well did we use available resources? • We will focus on the Web, but some of the concepts apply in other situations also.

13. Effectiveness • Two measures: • Precision • Of the results returned, what percentage are meaningful to the goal of the query? • Recall • Of the materials available that match the query, what percentage were returned? • Ex. Search returns 590,000 responses and 195 are relevant. How well did we do? • Not enough information. • Did the 590,000 include all relevant responses? If so, recall is perfect. • We have no way of knowing whether or not this is all the relevant resources • What is important is whether the right response is there. • 195/590,000 is not good precision!

14. User Web spider Search Indexer The Web Basic structure of Web Search Source: http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html

15. What is involved in text retrieval

16. The process Query entered Results Ranked Query Interpreted Index searched Items retrieved

17. The Collection • Where does the collection come from? • How is the index created? • Those are important distinguishing characteristics • Inverted Index -- Ordered list of terms related to the collected materials. Each term has an associated pointer to the related material(s). • www.cs.cityu.edu.hk/~deng/5286/T51.doc

18. An example: Unstructured data - circa 1650 • Which plays of Shakespeare contain the words BrutusANDCaesar but NOTCalpurnia? • One could grep all of Shakespeare’s plays for Brutus and Caesar, then strip out lines containing Calpurnia? • Slow (for large corpora) • NOTCalpurnia is non-trivial • Other operations (e.g., find the word Romans nearcountrymen) not feasible • Ranked retrieval (best documents to return) Source: http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html

19. A table showing incidence of the search terms in the documents 1 means the word is in the document What are the entries in the rows for the query: Brutus AND Caesar NOT Calpurnia Source: http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html

20. The vectors • Brutus: 1 1 0 1 0 0 • Caesar: 1 1 0 1 1 1 • Calpurnia: 0 1 0 0 0 0 • NOT Calpurnia: 1 0 1 1 1 1 • Brutus AND Caesar AND NOT Calpurnia: • AND them: 1 0 0 1 0 0 • Plays # 1 and 4 satisfy the query

21. The query terms in the documents Antony and Cleopatra, Act III, Scene ii Agrippa [Aside to DOMITIUS ENOBARBUS]: Why, Enobarbus, When Antony found Julius Caesar dead, He cried almost to roaring; and he wept When at Philippi he found Brutus slain. Hamlet, Act III, Scene ii Lord Polonius: I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. Source: http://www.stanford.edu/class/cs276/cs276-2006-syllabus.html

22. 2 4 8 16 32 64 128 1 2 3 5 8 13 21 34 Inverted index • For each term T, we must store a list of all documents that contain T. • Do we use an array or a list for this? Brutus Calpurnia Caesar 13 16 What happens if the word Caesar is added to document 14?

23. Brutus Calpurnia Caesar Dictionary Postings lists Inverted index • Linked lists generally preferred to arrays • Dynamic space allocation • Insertion of terms into documents easy • Space overhead of pointers Posting 2 4 8 16 32 64 128 1 2 3 5 8 13 21 34 13 16 Sorted by docID (more later on why).

24. Tokenizer Friends Romans Countrymen Token stream. Linguistic modules More on these later. friend friend roman countryman Modified tokens. roman Indexer 2 4 countryman 1 2 Inverted index. 16 13 Inverted index construction Documents to be indexed. Friends, Romans, countrymen.

25. Indexer steps • Sequence of (Modified token, Document ID) pairs. Doc 1 Doc 2 I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious

26. Sort by terms. Core indexing step.

27. Multiple term entries in a single document are merged. • Frequency information is added. Why frequency? Will discuss later.

28. The result is split into a Dictionary file and a Postings file.

29. Where do we pay in storage? Terms Pointers

30. 2 4 8 16 32 64 1 2 3 5 8 13 21 Query processing: AND • Consider processing the query: BrutusANDCaesar • Locate Brutus in the Dictionary; • Retrieve its postings. • Locate Caesar in the Dictionary; • Retrieve its postings. • “Merge” the two postings: 128 Brutus Caesar 34

31. Brutus Caesar 13 128 2 2 4 4 8 8 16 16 32 32 64 64 8 1 1 2 2 3 3 5 5 8 8 21 21 13 34 The merge • Walk through the two postings simultaneously, in time linear in the total number of postings entries 128 2 34 If the list lengths are x and y, the merge takes O(x+y) operations. Crucial: postings sorted by docID.

32. Boolean queries: Exact match • The Boolean Retrieval model is being able to ask a query that is a Boolean expression: • Boolean Queries are queries using AND, OR and NOT to join query terms • Views each document as a set of words • Is precise: document matches condition or not. • Primary commercial retrieval tool for 3 decades. • Professional searchers (e.g., lawyers) still like Boolean queries: • You know exactly what you’re getting.

33. Example: WestLaw http://www.westlaw.com/ • Largest commercial (paying subscribers) legal search service (started 1975; ranking added 1992) • Tens of terabytes of data; 700,000 users • Majority of users still use boolean queries • Example query: • What is the statute of limitations in cases involving the federal tort claims act? • LIMIT! /3 STATUTE ACTION /S FEDERAL /2 TORT /3 CLAIM • /3 = within 3 words, /S = in same sentence

34. Example: WestLaw http://www.westlaw.com/ • Another example query: • Requirements for disabled people to be able to access a workplace • disabl! /p access! /s work-site work-place (employment /3 place • Note that SPACE is disjunction, not conjunction! • Long, precise queries; proximity operators; incrementally developed; not like web search • Professional searchers often like Boolean search: • Precision, transparency and control • But that doesn’t mean they actually work better….

35. Boolean queries: More general merges • Exercise: Adapt the merge for the queries: BrutusAND NOTCaesar BrutusOR NOTCaesar

36. 2 4 8 16 32 64 128 1 2 3 5 8 16 21 34 Query optimization • What is the best order for query processing? • Consider a query that is an AND of t terms. • For each of the t terms, get its postings, then AND them together. Brutus Calpurnia Caesar 13 16 Query: BrutusANDCalpurniaANDCaesar

37. Brutus 2 4 8 16 32 64 128 Calpurnia 1 2 3 5 8 13 21 34 Caesar 13 16 Query optimization example • Process in order of increasing freq: • start with smallest set, then keepcutting further. This is why we kept freq in dictionary Execute the query as (CaesarANDBrutus)ANDCalpurnia.

38. What’s ahead in IR?Beyond term search • What about phrases? • Stanford University • Proximity: Find GatesNEAR Microsoft. • Need index to capture position information in docs. More later. • Zones in documents: Find documents with (author = Ullman) AND (text contains automata).

39. Evidence accumulation • 1 vs. 0 occurrence of a search term • 2 vs. 1 occurrence • 3 vs. 2 occurrences, etc. • Usually more seems better • Need term frequency information in docs

40. Ranking search results • Boolean queries give inclusion or exclusion of docs. • Often we want to rank/group results • Need to measure proximity from query to each doc. • Need to decide whether docs presented to user are singletons, or a group of docs covering various aspects of the query.

41. IR vs. databases:Structured vs unstructured data • Structured data tends to refer to information in “tables” Employee Manager Salary Smith Jones 50000 Chang Smith 60000 Ivy Smith 50000 Typically allows numerical range and exact match (for text) queries, e.g., Salary < 60000 AND Manager = Smith.

42. Unstructured data • Typically refers to free text • Allows • Keyword queries including operators • More sophisticated “concept” queries e.g., • find all web pages dealing with drug abuse • Classic model for searching text documents

43. Semi-structured data • In fact almost no data is “unstructured” • E.g., this slide has distinctly identified zones such as the Title and Bullets • Facilitates “semi-structured” search such as • Title contains data AND Bullets contain search … to say nothing of linguistic structure

44. More sophisticated semi-structured search • Title is about Object Oriented Programming AND Author something like stro*rup • where * is the wild-card operator • Issues: • how do you process “about”? • how do you rank results? • The focus of XML search.

45. Clustering and classification • Given a set of docs, group them into clusters based on their contents. • Given a set of topics, plus a new doc D, decide which topic(s) D belongs to.

46. The web and its challenges • Unusual and diverse documents • Unusual and diverse users, queries, information needs • Beyond terms, exploit ideas from social networks • link analysis, clickstreams ... • How do search engines work? And how can we make them better?

47. More sophisticated information retrieval • Cross-language information retrieval • Question answering • Summarization • Text mining • …

48. Crawling the web • Misnomer as the spider or robot does not actually move about the web • Program sends a normal request for the page, just as a browser would. • Retrieve the page and parse it. • Look for anchors -- pointers to other pages. • Put them on a list of URLs to visit • Extract key words (possibly all words) to use as index terms related to that page • Take the next URL and do it again • Actually, the crawling and processing are parallel activities

49. Responding to search queries • Use the query string provided • Form a boolean query • Join all words with AND? With OR? • Find the related index terms • Return the information available about the pages that correspond to the query terms. • Many variations on how to do this. Usually proprietary to the company.

50. Making the connections • Stemming • Making sure that simple variations in word form are recognized as equivalent for the purpose of the search: exercise, exercises, exercised, for example. • Indexing • A keyword or group of selected words • Any word (more general) • How to choose the most relevant terms to use as index elements for a set of documents. • Build an inverted file for the chosen index terms.