Lecture 16: IR Components 2

1. Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00 pm Spring 2007 http://courses.ischool.berkeley.edu/i240/s07 Lecture 16: IR Components 2 Principles of Information Retrieval

2. Overview • Review • IR Components • Text Processing and Stemming • Relevance Feedback

3. Stemming and Morphological Analysis • Goal: “normalize” similar words • Morphology (“form” of words) • Inflectional Morphology • E.g,. inflect verb endings and noun number • Never change grammatical class • dog, dogs • tengo, tienes, tiene, tenemos, tienen • Derivational Morphology • Derive one word from another, • Often change grammatical class • build, building; health, healthy

4. Simple “S” stemming • IF a word ends in “ies”, but not “eies” or “aies” • THEN “ies”  “y” • IF a word ends in “es”, but not “aes”, “ees”, or “oes” • THEN “es” “e” • IF a word ends in “s”, but not “us” or “ss” • THEN “s”  NULL Harman, JASIS 1991

5. Stemmer Examples

6. Errors Generated by Porter Stemmer (Krovetz 93)

7. Automated Methods • Stemmers: • Very dumb rules work well (for English) • Porter Stemmer: Iteratively remove suffixes • Improvement: pass results through a lexicon • Newer stemmers are configurable (Snowball) • Powerful multilingual tools exist for morphological analysis • PCKimmo, Xerox Lexical technology • Require a grammar and dictionary • Use “two-level” automata • Wordnet “morpher”

8. Wordnet • Type “wn word” on irony. • Large exception dictionary: • Demo aardwolves aardwolf abaci abacus abacuses abacus abbacies abbacy abhenries abhenry abilities ability abkhaz abkhaz abnormalities abnormality aboideaus aboideau aboideaux aboideau aboiteaus aboiteau aboiteaux aboiteau abos abo abscissae abscissa abscissas abscissa absurdities absurdity …

9. Using NLP • Strzalkowski (in Reader) Text NLP repres Dbase search TAGGER PARSER TERMS NLP:

10. Using NLP INPUT SENTENCE The former Soviet President has been a local hero ever since a Russian tank invaded Wisconsin. TAGGED SENTENCE The/dt former/jj Soviet/jj President/nn has/vbz been/vbn a/dt local/jj hero/nn ever/rb since/in a/dt Russian/jj tank/nn invaded/vbd Wisconsin/np ./per

11. Using NLP TAGGED & STEMMED SENTENCE the/dt former/jj soviet/jj president/nn have/vbz be/vbn a/dt local/jj hero/nn ever/rb since/in a/dt russian/jj tank/nn invade/vbd wisconsin/np ./per

12. Using NLP PARSED SENTENCE [assert [[perf [have]][[verb[BE]] [subject [np[n PRESIDENT][t_pos THE] [adj[FORMER]][adj[SOVIET]]]] [adv EVER] [sub_ord[SINCE [[verb[INVADE]] [subject [np [n TANK][t_pos A] [adj [RUSSIAN]]]] [object [np [name [WISCONSIN]]]]]]]]]

13. Using NLP EXTRACTED TERMS & WEIGHTS President 2.623519 soviet 5.416102 President+soviet 11.556747 president+former 14.594883 Hero 7.896426 hero+local 14.314775 Invade 8.435012 tank 6.848128 Tank+invade 17.402237 tank+russian 16.030809 Russian 7.383342 wisconsin 7.785689

14. Same Sentence, different sys INPUT SENTENCE The former Soviet President has been a local hero ever since a Russian tank invaded Wisconsin. TAGGED SENTENCE (using uptagger from Tsujii) The/DT former/JJ Soviet/NNP President/NNP has/VBZ been/VBN a/DT local/JJ hero/NN ever/RB since/IN a/DT Russian/JJ tank/NN invaded/VBD Wisconsin/NNP ./.

15. Same Sentence, different sys CHUNKED Sentence (chunkparser – Tsujii) (TOP (S (NP (DT The) (JJ former) (NNP Soviet) (NNP President) ) (VP (VBZ has) (VP (VBN been) (NP (DT a) (JJ local) (NN hero) ) (ADVP (RB ever) ) (SBAR (IN since) (S (NP (DT a) (JJ Russian) (NN tank) ) (VP (VBD invaded) (NP (NNP Wisconsin) ) ) ) ) ) ) (. .) ) )

16. Same Sentence, different sys Enju Parser ROOT ROOT ROOT ROOT -1 ROOT been be VBN VB 5 been be VBN VB 5 ARG1 President president NNP NNP 3 been be VBN VB 5 ARG2 hero hero NN NN 8 a a DT DT 6 ARG1 hero hero NN NN 8 a a DT DT 11 ARG1 tank tank NN NN 13 local local JJ JJ 7 ARG1 hero hero NN NN 8 The the DT DT 0 ARG1 President president NNP NNP 3 former former JJ JJ 1 ARG1 President president NNP NNP 3 Russian russian JJ JJ 12 ARG1 tank tank NN NN 13 Soviet soviet NNP NNP 2 MOD President president NNP NNP 3 invaded invade VBD VB 14 ARG1 tank tank NN NN 13 invaded invade VBD VB 14 ARG2 Wisconsin wisconsin NNP NNP 15 has have VBZ VB 4 ARG1 President president NNP NNP 3 has have VBZ VB 4 ARG2 been be VBN VB 5 since since IN IN 10 MOD been be VBN VB 5 since since IN IN 10 ARG1 invaded invade VBD VB 14 ever ever RB RB 9 ARG1 since since IN IN 10

17. Assumptions in IR • Statistical independence of terms • Dependence approximations

18. Statistical Independence Two events x and y are statistically independent if the product of their probability of their happening individually equals their probability of happening together.

19. Statistical Independence and Dependence • What are examples of things that are statistically independent? • What are examples of things that are statistically dependent?

20. Statistical Independence vs. Statistical Dependence • How likely is a red car to drive by given we’ve seen a black one? • How likely is the word “ambulence” to appear, given that we’ve seen “car accident”? • Color of cars driving by are independent (although more frequent colors are more likely) • Words in text are not independent (although again more frequent words are more likely)

21. Lexical Associations • Subjects write first word that comes to mind • doctor/nurse; black/white (Palermo & Jenkins 64) • Text Corpora yield similar associations • One measure: Mutual Information (Church and Hanks 89) • If word occurrences were independent, the numerator and denominator would be equal (if measured across a large collection)

22. Interesting Associations with “Doctor” (AP Corpus, N=15 million, Church & Hanks 89)

23. Un-Interesting Associations with “Doctor” These associations were likely to happen because the non-doctor words shown here are very common and therefore likely to co-occur with any noun.

24. Today • Relevance Feedback • aka query modification • aka “more like this”

25. IR Components • A number of techniques have been shown to be potentially important or useful for effective IR (in TREC-like evaluations) • Today and over the next couple weeks (except for Spring Break) we will look at these components of IR systems and their effects on retrieval • These include: Relevance Feedback, Latent Semantic Indexing, clustering, and application of NLP techniques in term extraction and normalization

26. Querying in IR System Storage Line Interest profiles & Queries Documents & data Information Storage and Retrieval System Rules of the game = Rules for subject indexing + Thesaurus (which consists of Lead-In Vocabulary and Indexing Language Formulating query in terms of descriptors Indexing (Descriptive and Subject) Storage of profiles Storage of Documents Store1: Profiles/ Search requests Store2: Document representations Comparison/ Matching Potentially Relevant Documents

27. Relevance Feedback in an IR System Storage Line Interest profiles & Queries Documents & data Information Storage and Retrieval System Rules of the game = Rules for subject indexing + Thesaurus (which consists of Lead-In Vocabulary and Indexing Language Formulating query in terms of descriptors Indexing (Descriptive and Subject) Storage of profiles Storage of Documents Store1: Profiles/ Search requests Store2: Document representations Comparison/ Matching Potentially Relevant Documents Selected relevant docs

28. Query Modification • Changing or Expanding a query can lead to better results • Problem: how to reformulate the query? • Thesaurus expansion: • Suggest terms similar to query terms • Relevance feedback: • Suggest terms (and documents) similar to retrieved documents that have been judged to be relevant

29. Relevance Feedback • Main Idea: • Modify existing query based on relevance judgements • Extract terms from relevant documents and add them to the query • and/or re-weight the terms already in the query • Two main approaches: • Automatic (psuedo-relevance feedback) • Users select relevant documents • Users/system select terms from an automatically-generated list

30. Relevance Feedback • Usually do both: • Expand query with new terms • Re-weight terms in query • There are many variations • Usually positive weights for terms from relevant docs • Sometimes negative weights for terms from non-relevant docs • Remove terms ONLY in non-relevant documents

31. Rocchio Method

32. Rocchio/Vector Illustration Information 1.0 D1 Q’ 0.5 Q0 Q” D2 0 0.5 1.0 Retrieval Q0 = retrieval of information = (0.7,0.3) D1 = information science = (0.2,0.8) D2 = retrieval systems = (0.9,0.1) Q’ = ½*Q0+ ½ * D1 = (0.45,0.55) Q” = ½*Q0+ ½ * D2 = (0.80,0.20)

33. Example Rocchio Calculation Relevant docs Non-rel doc Original Query Constants Rocchio Calculation Resulting feedback query

34. Rocchio Method • Rocchio automatically • re-weights terms • adds in new terms (from relevant docs) • have to be careful when using negative terms • Rocchio is not a machine learning algorithm • Most methods perform similarly • results heavily dependent on test collection • Machine learning methods are proving to work better than standard IR approaches like Rocchio

35. Probabilistic Relevance Feedback Document Relevance Given a query term t + - + r n-r n - R-r N-n-R+r N-n R N-R N Document indexing Where N is the number of documents seen Robertson & Sparck Jones

36. Robertson-Spark Jones Weights • Retrospective formulation --

37. Robertson-Sparck Jones Weights Predictive formulation

38. Using Relevance Feedback • Known to improve results • in TREC-like conditions (no user involved) • So-called “Blind Relevance Feedback” typically uses the Rocchio algorithm with the assumption that the top N documents in an initial retrieval are relevant • What about with a user in the loop? • How might you measure this? • Let’s examine a user study of relevance feedback by Koenneman & Belkin 1996.

39. Questions being InvestigatedKoenemann & Belkin 96 • How well do users work with statistical ranking on full text? • Does relevance feedback improve results? • Is user control over operation of relevance feedback helpful? • How do different levels of user control effect results?

40. How much of the guts should the user see? • Opaque (black box) • (like web search engines) • Transparent • (see available terms after the r.f. ) • Penetrable • (see suggested terms before the r.f.) • Which do you think worked best?

42. Penetrable… Terms available for relevance feedback made visible(from Koenemann & Belkin)

43. Details on User StudyKoenemann & Belkin 96 • Subjects have a tutorial session to learn the system • Their goal is to keep modifying the query until they’ve developed one that gets high precision • This is an example of a routing query (as opposed to ad hoc) • Reweighting: • They did not reweight query terms • Instead, only term expansion • pool all terms in rel docs • take top N terms, where • n = 3 + (number-marked-relevant-docs*2) • (the more marked docs, the more terms added to the query)

44. Details on User StudyKoenemann & Belkin 96 • 64 novice searchers • 43 female, 21 male, native English • TREC test bed • Wall Street Journal subset • Two search topics • Automobile Recalls • Tobacco Advertising and the Young • Relevance judgements from TREC and experimenter • System was INQUERY (Inference net system using (mostly) vector methods)

45. Sample TREC query

46. Evaluation • Precision at 30 documents • Baseline: (Trial 1) • How well does initial search go? • One topic has more relevant docs than the other • Experimental condition (Trial 2) • Subjects get tutorial on relevance feedback • Modify query in one of four modes • no r.f., opaque, transparent, penetration

47. Precision vs. RF condition (from Koenemann & Belkin 96)

48. Effectiveness Results • Subjects with R.F. did 17-34% better performance than no R.F. • Subjects with penetration case did 15% better as a group than those in opaque and transparent cases.

49. Number of iterations in formulating queries (from Koenemann & Belkin 96)

50. Number of terms in created queries (from Koenemann & Belkin 96)