1. Personalizing Web Search Jaime Teevan, MIT with Susan T. Dumais and Eric Horvitz, MSR

3. Demo

4. Personalizing Web Search • Motivation • Algorithms • Results • Future Work

5. Personalizing Web Search • Motivation • Algorithms • Results • Future Work

6. Study of Personal Relevancy • 15 SIS users x ~10 queries • Evaluate 50 results • Highly relevant / Relevant / Irrelevant • Query selection • Previously issued query • Chose from 10 pre-selected queries • Collected evaluations for 137 queries • 53 of pre-selected queries (2-9/query)

7. Relevant Results Have Low Rank Highly Relevant Relevant Irrelevant

8. Same Query, Different Intent • Different meanings • “Information about the astronomical/astrological sign of cancer” • “information about cancer treatments” • Different intents • “is there any new tests for cancer?” • “information about cancer treatments”

9. Same Intent, Different Evaluation • Query: Microsoft • “information about microsoft, the company” • “Things related to the Microsoft corporation” • “Information on Microsoft Corp” • 31/50 rated as not irrelevant • Only 6/31 do more than one agree • All three agree only for www.microsoft.com

10. More to Understand • Do people cluster? • Even if they can’t state their intention • How are the differences reflected? • Can they be seen from the information on a person’s computer? • Can we do better than the ranking that would make everyone the most happy? • Best common ranking: +38% • Best personalized ranking: +55%

11. Personalizing Web Search • Motivation • Algorithms • Results • Future Work

12. Personalization Algorithms • Standard IR • Related to relevance feedback • Query expansion Query Server Document Client User v. Result re-ranking

13. Result Re-Ranking • Takes full advantage of SIS • Ensures privacy • Good evaluation framework • Look at light weight user models • Collected on server side • Sent as query expansion

14. BM25 with Relevance Feedback Score = Σtfi * wi N ni R ri N ni wi = log

15. BM25 with Relevance Feedback Score = Σtfi * wi N ni R ri (ri+0.5)(N-ni-R+ri+0.5) (ni-ri+0.5)(R-ri+0.5) wi = log

16. User Model as Relevance Feedback Score = Σtfi * wi N R N’ = N+R ni’ = ni+ri ri ni (ri+0.5)(N-ni-R+ri+0.5) (ni- ri+0.5)(R-ri+0.5) (ri+0.5)(N’-ni’-R+ri+0.5) (ni’- ri+0.5)(R-ri+0.5) wi = log

17. User Model as Relevance Feedback World Score = Σtfi * wi N User R ri ni

18. User Model as Relevance Feedback World Score = Σtfi * wi N User World related to query R ri ni ni N

19. User Model as Relevance Feedback World Score = Σtfi * wi N User World related to query R ri ni R ni N User related to query ri Query Focused Matching

20. User Model as Relevance Feedback World Focused Matching World Score = Σtfi * wi N User Web related to query R ri ni R ni N User related to query ri Query Focused Matching

21. Parameters • Matching • User representation • World representation • Query expansion

22. Parameters • Matching • User representation • World representation • Query expansion Query focused World focused

23. Parameters • Matching • User representation • World representation • Query expansion Query focused World focused

24. User Representation • Stuff I’ve Seen (SIS) index • Recently indexed documents • Web documents in SIS index • Query history • Relevance judgments • None

25. Parameters • Matching • User representation • World representation • Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None

26. Parameters • Matching • User representation • World representation • Query expansion Query Focused World Focused All SIS Recent SIS Web SIS Query History Relevance Feedback None

27. World Representation • Document Representation • Full text • Title and snippet • Corpus Representation • Web • Result set – title and snippet • Result set – full text

28. Parameters • Matching • User representation • World representation • Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None Full text Title and snippet Web Result set – full text Result set – title and snippet

29. Parameters • Matching • User representation • World representation • Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None Full text Title and snippet Web Result set – full text Result set – title and snippet

30. Query Expansion • All words in document • Query focused The American Cancer Society is dedicated to eliminating cancer as a major health problem by preventing cancer, saving lives, and diminishing suffering through ... The American Cancer Society is dedicated to eliminating cancer as a major health problem by preventing cancer, saving lives, and diminishing suffering through ...

31. Parameters • Matching • User representation • World representation • Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None Full text Title and snippet Web Result set – full text Result set – title and snippet All words Query focused

32. Parameters • Matching • User representation • World representation • Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None Full text Title and snippet Web Result set – full text Result set – title and snippet All words Query focused

33. Parameters • Matching • User representation • World representation • Query expansion Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None Full text Title and snippet Web Result set – full text Result set – title and snippet All words Query focused

34. Personalizing Web Search • Motivation • Algorithms • Results • Future Work

35. Baselines • Best possible • Random • Text based ranking • Web ranking • URL Boost http://mail.yahoo.com/inbox/msg10 http://mail.yahoo.com/inbox/msg10 +1 http://mail.yahoo.com/inbox/msg10 +1

36. Best Parameter Settings • Richer user representation better • SIS > Recent > Web > Query History > None Suggests rich client important • Efficiency hacks don’t hurt • Snippets query focused • Length normalization not an issue • Query focus good

37. Text Alone Not Enough • Better than some baselines • Better than random • Better than no user representation • Better than relevance feedback • Worse than Web results • Blend in other features • Web ranking • URL boost

38. Good, but Lots of Room to Grow • Best combination: 9.1% improvement • Best possible: 51.5% improvement • Assumes best Web combination selected • Only improves results 2/3 of the time

39. Personalizing Web Search • Motivation • Algorithms • Results • Future Work

40. Finding the Best Parameter Setting • Almost always some parameter setting that improves results • Use learning to select parameters • Based on individual • Based on query • Based on results • Give user control?

41. Further Exploration of Algorithms • Larger parameter space to explore • More complex user model subsets • Different parsing (e.g., phrases) • Tune BM25 parameters • What is really helping? • Generic user model or personal model • Use different indices for the queries • Deploy system

42. Practical Issues • Efficiency issues • Can interfaces mitigate some of the issues? • Merging server and client • Query expansion • Get more relevant results in the set to be re-ranked • Design snippets for personalization

43. Thank you!