Mediated Information Retrieval – The WebCluster and MIR Projects –

1. Mediated Information Retrieval– The WebCluster and MIR Projects – Gheorghe Muresan School of Communication, Information and Library Sciences Rutgers University

2. Structure of the talk • The WebCluster project • Design decisions in WebCluster • The MIR project • Integrated approach to interaction modeling, logging and analysis

3. Information Need WebCluster - Motivation (within some subject domain) • Gulfs • information need query • structured subject domain unstructured target collection (WWW) WWW_SearchEngine Search engine Domain Query

4. Interaction in the library Information need 2. Consult catalog 1. Select library Information Need Formulation 3. Browse shelves 4. Use inter-library scheme

5. WWW Results Information Need Formulation Results Can we simulate the library interaction ? Structured source collections Information need 3. Search WWW 1. Select source collection 2. Explore source collection with ClusterBook

6. Information need Specialised source The mediated access interaction Topical documents Web search engine WebCluster Query Target collection (WWW)

7. Interaction model vs. prototype • Structuring the source collection • Document clustering • Supervised classification • Manual (intellectual) classification • Exploring the structured source collection • Metaphor – Library, book, encyclopaedia • Visualization tool – Folder metaphor, hyperbolic tree, themescape, cone trees, thematic maps • Search strategies supported – Best match or cluster-based searching, browsing

8. Model vs. prototype • Interaction model • Explicit(the user marks relevant documents) vs. implicit(cues on relevance are derived based on user behavior/actions) • Transparent(the user is aware) vs. opaque(the user is happy to see effect of ‘magic’) • Automatic vs. manual/intellectual generation of the mediated query • Query model • Language models (generative, Kullback-Leibler) • Probabilistic models • Rocchio or other RF-specific formulae

9. ClusterBook - Source collection

10. ClusterBook - Target collection

11. Informal experiments - Objectives - • Test the users’ reaction to the mediated access concept • Test the user satisfaction regarding the functionality of the system, and the relevance of the documents retrieved • Formative usability testing - some volunteers were not only experienced searchers, but also had experience in evaluating IR systems • Comparison of user generated queries vs. system generated queries • Note. These experiments were run at different stages of the development

12. Informal experiments - Experimental procedure - • Subjects received introduction to the system • Task assigned: “You are a trainee in a newspaper. You support the journalists by providing information for the topic of their articles.” • Sample topics: • The history of the Brasilian debt crisis • How are the quotas for growing coffee set and controlled on a world-wide basis ? • Source collection: a sub-collection of Reuters (newspaper articles) • Steps followed by users (explicit scenario): • Formulate a query and record it • Browse source collection, select ‘best’ cluster, edit query generated by system, submit it to the search engine • Submit to the same search engine the initial, self-generated query • Compare results of the two searches

13. Informal experiments - Results - • Users found the mediation useful for unfamiliar topics • The system nearly always proposed new, good query terms • Users not always good at recognizing ‘good’ query terms • The system proposed bad query terms (not specific to the topic) • the opaque scenario not viable unless the query formulation is improved • The two-step process was questioned when: • the query formulation was considered easy, for a familiar topic • the documents of the source collection were considered sufficient to cover the information need • Complete link, group average – OK; single link – bad • Overall, the system is usable

14. Consequences of informal experiments • Formal experiments are needed to verify the main assumptions: • The Cluster Hypothesis holds for a specialized collection • Good clusters can be found with the search strategies provided • Mediated queries can improve retrieval effectiveness • The effect on retrieval performance of various parameters should be compared • Weighting schemes • Clustering methods • Search strategies

15. Portable Generators ... Critical issue: The label generation Wind Energy • Document representatives • searching • Cluster representatives • browsing • searching • mediation • Collection representatives • collection selection ... Power Generation Propulsion Fixed Plants Coastal Wind Farms Inland Wind Farms Design of …. Wind generators for yachts Pacific Rim Wind Farms Design of Coastal Wind Farms Desert Wind Farms

16. Cluster-based mediation (realistic mediation) Source collection Target collection Mediation experiment - simulations • Objectives: • Test the potential of mediation to increase retrieval effectiveness • Test the effect on effectiveness of a variety of parameters Search engine Search engine Topic-based mediator (upperbound) Simple query generator (baseline)

17. Experimental setup • Interactive track of TREC-8 • Offers relevance judgments for complex topics, with a multitude of aspects • Offers the experimental design for the user experiment • Six topics with 12 to 56 aspects each • Target collection: FT 1991-4, with 210,158 articles • Source collection built based on relevance judgments: half of the relevant documents, their nearest neighbors, plus the documents judged non-relevant

18. Results – the cluster hypothesis • Aspectual cluster hypothesis confirmed by an extended version of the van Rijsbergen – Sparck Jones separation test • Similarity between pairs of docs covering the same aspect is higher than between pairs of docs covering the same topics, which is higher than between pairs of docs in the collection • Consequence confirmed: clustering groups documents in pockets of relevance

19. Results – retrieval effectiveness • Tf-Idf > KL > RelFreq as weighting schemes for document representation • Adding disambiguation terms to the query increases recall, but decreases precision • Nearest-neighbor mediation (“more like this”) highly significantly improves both recall and precision, even if just one exemplary document is offered for each topic aspect • Cosine and Dice performs similarly

20. Mediation results • Upperbound experiment (all relevant docs known in source) • Both recall and precision increase with query length • Query term weights strongly affect performance • No evidence that uniformity of term frequency affects performance • Clustered source mediation • Best cluster mediation increases P, decreases R • “Fuse and search” – strong increase in R and P • “Search and fuse” – good R, terrible P !

21. Contributions of WebCluster • Proposes and explores system-based mediated access to very large heterogeneous document collections • Explores the use of clustering for capturing the topical, semantic structure of a problem domain (as represented by a specialized collection) • Explores the use of language models for building cluster and document representatives • Offers a framework for building structured portals on the WWW • Offers a framework for building collaborative environments

22. Structure of the talk • The WebCluster project • Design decisions in WebCluster • The MIR project • Integrated approach to interaction modeling, logging and analysis

23. Structure of the talk • The WebCluster project • Design decisions in WebCluster • The MIR project • Integrated approach to interaction modeling, logging and analysis

24. User experiment – effectiveness of mediated information retrieval for Web searches • NL- Non-mediated and Linear, NC- Non-mediated and Combination • ML- Mediated and Linear, MC- Mediated and Combination

25. Research Hypotheses • The mediated system is conducive to higher effectiveness than the non-mediated system • The combination of linear/ranked display with a hierarchic/clustered display is conducive to higher effectiveness than simple ranked display

26. Mediation assumptions • Relevant documents tend to be clustered together in the source collection • The cluster hypotheses • Subjects can identify relevant documents • Subjects spend time exploring the source collection and some relevant documents • Queries submitted after mediation are better • Longer, higher clarity, wider vocabulary

27. Other areas of exploration • Interaction models • Transitions between states and activities • The effect on search behavior of subject expertise or familiarity with a topic • The subjects’ ability to recognize good documents and clusters based on snippets or labels • Compare user-generated queries with system-generated queries in terms of performance

28. User experiment – no mediation

29. User experiment – mediated access

30. User experiment – mediated access

31. Structure of the talk • The WebCluster project • Design decisions in WebCluster • The MIR project • Integrated approach to interaction modeling, logging and analysis

32. Motivation • Interest in studying Human Information Behavior and Interactive Information Retrieval • Qualitative aspects • Patterns of behavior → User models → Predictions of future behavior • Quantitative aspects • # queries, # query terms, # documents viewed / opened / saved, # errors / corrections, time spent → Conclusions regarding retrieval effectiveness & efficiency • Typical tools • Think-aloud protocols, video recording, questionnaires, interviews, activity logging

33. Motivation • Logging – options: • Commercial tools (Morae, uLog, Camtasia, etc) • Expensive, less control over what is logged, format usually proprietary • DIY – log events related to the research questions • Rather inflexible – what if new research ideas come to light? • Idea #1: log all semantic events • Identified during interaction / interface design→ integrate: • Interface design • Logger design • Log analyzer design

34. Motivation - practical • Frustration with existing practices in IR research • Rutgers participation in Interactive TREC 2002 • User interface • Logging • Idea #2: use state-based design, logging and analysis • Advantages: • Design tools are plentiful • The entire research team can participate in design • Once the design is completed, the procedures to generate the logging software and the log analyzer are deterministic

35. Prepare experimental system Design system Build system Add functionality for logging interactions Typical interactive IR experiment Research Hypotheses Analyze experimental data Draw conclusions Run experiment Generate experimental data, including logs of interactions

36. Problems with this experimental model(based on anectodal evidence) • The system is built and the extraction of experimental data from logs is done by “those who can program” in the research group • Most researchers are not involved in specifying system requirements • The design stage is often skipped • The system may display usability problems, which affects the exploration of the research problems • The logging functionality is added ad-hoc, in unsystematic fashion • There is no standard format for interaction logs; additional software is needed for parsing logs and extracting useful data

37. Build system XML logger Experimental system Proposed integrated approach(Keywords: UML, DTD, XML) Research Hypotheses Conceptual design of the system (UML model of interactions) Data visualization DTD model of interactions Analyze experimental data XML parser XML log format Run experiment Generate experimental data, including logs of interactions

38. Summary • Model the interaction using UML diagrams • This allows the whole research team to contribute to the design of the user interfaces, and supports the documentation of the interface. • Derive a DTD coding for the states of the user interface, the valid user actions in every state and the state transitions that take place based on user actions; • Use XML to log the user actions and their outcomes based on the interaction DTD; • Based on the DTD, generate a log parser for the log analysis; • The log analysis provides interaction information and can also be used for generating a visual / graphic representation of the interaction

39. MIR – state diagram

40. MIR – DTD

41. MIR – XML log

42. MIR – logger class diagram

43. Design decisions • Design patterns • State – each state of the interface/system is modeled by a class • Inheritance (class hierarchy) is used to model sub-states (states at different levels of granularity) • Composition is used to model orthogonal states • Visitor – decouples the strategy chosen for parsing / visiting the log from the actions taken in each node • Strategy – supports different analysis strategies • DOMLogAnalyzer – visits the entire log tree, for a comprehensive analysis • XPathAnalyzer – visits only a selection of nodes, relevant for a certain RH (“log/record/message/SaveDoc”)

44. Design decisions • Singletons vs. multiple objects for states • Singleton – one object for each class • Adv: simple • Disadv: only appropriate for cumulative data or summaries • Multiple objects • Adv: supports accurate, detailed analysis • Explicit vs. implicit logging of states • Explicit: allows a human reader to interpret the logs; redundancy; problems capturing orthogonal states • Implicit: only events are captured, states are re-created

45. Types of analysis supported Think 4 EditQuery 9 ViewTargetHitList 15 ViewTargetDoc 78 SavingDoc 16 ViewTargetHitList 6 ViewTargetDoc 31 ViewTargetDoc 9 ViewTargetDoc 35 SavingDoc 11 ViewTargetHitList 3 ViewTargetDoc 173 SavingDoc 16 ViewTargetHitList 14 EditQuery 7 ViewTargetHitList 4 ViewTargetDoc 17 ViewTargetDoc 59 ViewTargetDoc 51 ViewTargetDoc 39 EditQuery 13 ViewTargetHitList 25 ViewTargetDoc 38 SavingDoc 15 … • State transitions, user behavior • Average user vs. individual user • Levels of state granularity • (Think, EditQuery, ViewResults (ExploreSource (ViewSourceHierarchy, ViewSourceHitList, ViewSourceDoc), ExploreTarget (ViewTargetHitList, ViewTargetDoc, ViewSavedDoc))) • Statistical analysis on qual data • ANOVA shows no difference in number of saved docs between non-mediated condition (m=3.94, sd=1.76) and mediated condition (m=3.13, sd=1.62)

46. Conclusions • Advantages of the proposed method • Better teamwork • All members contribute and are responsible for the design • More accurate experimental results • Increased usability of the experimental system • Accurate data, due to accurate logging of events • Less effort in testing and debugging, as well as in parsing and analyzing results • DTD offers the interaction template • XML logs support debugging • Available open-source XML parsers

47. Future work • Automatic (or semi-automatic) generation of the DTD model from the UML model • Conceptual problem: designing a transition scheme between the two models • Practical problem: interpreting the format that various modeling packages use to store UML models • Visualization of the interaction • Model: timeline of the interaction vs. summary • Format: HTML vs. SVG vs. … • Automatic generation: programming language (Java) vs. transformation template (XSLT)

48. Questions ?

49. Query formulation problems • Vague information need • Vocabulary mismatch • Difficulty of query language syntax • Lack of context, ambiguity of terms • Lack of a search strategy • No understanding of the underlying indexing/searching model • Note. TREC experiments have shown that the quality of the query has a higher impact on retrieval effectiveness than weighting schemes or search algorithms.

50. Science ... Computing, Mathematics Physics Computing Computer Programming language Mathematics ... Screen Algebra Pascal C++ Keyboard Role of structure • Reveals the semantic structure of the domain & its concepts • Groups (semantically ?) similar documents • Supports exploration and concept formation • Supports term disambiguation (context) • (Has potential for efficient retrieval) • (Has potential for effective retrieval)