(Spoken) Dialogue and Information Retrieval

(Spoken) Dialogueand Information Retrieval Antoine Raux Dialogs on Dialogs Group 10/24/2003

Outline • Interactive Information Retrieval Systems (Belkin et al) • EUREKA: Dialogue-based IR for Low Bandwidth Devices • Voice Access to IR

Cases, Scripts, and Information-Seeking Strategies • Belkin, Cool (Rutgers)Stein, Thiel (GMD-IPSI) • Long journal article (1995) • From the IR community (Expert Systems)

IR as Interaction • Traditional IR research focuses on document/query representation and comparison • Need to focus on the user • Represent IR as a dialogue between an information seeker and an information provider

Information-Seeking Strategies • Represent information-seeking behavior along 4 dimensions: • Method of Interaction (scanning vs searching) • Goal of Interaction (learning vs selecting) • Mode of Retrieval (recognition vs specification) • Resource Considered (information vs meta-info) • Binary values  16 strategies (ISS)

Dialogue Structures for Information Seeking • Mix of different formalisms: • Recursive state-based schemas (COR)e.g. Request  Promise  Inform  Be contented • Scripts: prototypical interaction for each ISS • Goal trees Retrieve Specified Items Specify Characteristic Recognize Desired Items Offer choice Select and Specify

Deriving Scripts from Data • Case-based approach: problem solving using previously stored solved instances • Match a sequence of action to a state-based schema • Extract goal tree • Identify goal (which ISS?)

The MERIT System • Theory vs Practice… • Graphical interface (not NL dialogue) • User does case selection (for eventual case-based reasoning) • Example task is relational database (not free text IR): uses form filling (!)

Discussion • Contribution to IR: user-centered view, application of many non-IR theories (discourse, CBR) • BUT: too complicated for the user?

Discussion • Contribution to Dialogue Systems: difficult task (not often dealt with in DS), CBR (can we learn dialogue structure from data?) • BUT: lacks a good, unified, practical framework (too many different paradigms applied…)

Dialogue-based IR: Why? • Google-like interface still predominant (despite MERIT) • Why? • Users receives a lot of information (document titles, summaries) and use it as they want • Very simple to learn • Very flexible • BUT: works on high bandwidth devices

Dialogue-based IR: Why? • For low bandwidth devices (PDA, phone), information-rich interface don’t work • Only small pieces of information exchanged at a time • System has to select • Less information, more interaction

EUREKA: Idea • Use dialogue to submit queries to a web search engine, browse through the hierarchically clustered results, perform query reformulation/refinement, etc…

EUREKA: Overview • Backend: Vivisimo (through web scraper) • Dialogue Management: RavenClaw (successor of CMU Communicator) • Language Understanding: Light Open Vocabulary Parser • NLG/TTS: template-based & Festival

Backend: Vivisimo • Available clustering meta-search enginewww.vivisimo.com • Hand-written Perl web scraper (hope Vivisimo doesn’t change their page design by the end of the semester…)

LOV Parser • Problem: traditional NL parsers require a dictionary  not applicable to open domain IR • Solution (implemented in C++): • fix a small number of one-word commands (new_query, open, list_clusters) • parse each line as “[command] [arguments]” or “[command]” or “[arguments]”

EUREKA Greet User Prompt Query New Query Open Cluster Close Cluster … Submit Query Get Cluster List Get Doc List Inform Results Dialogue Management: RavenClaw • Hierarchical agent architecture:

NLG/TTS • Template-based Language Generation (e.g. “I found <n_doc> documents.”) • General purpose Festival voice for TTS NB: Browsing through lists is not efficient with speech, even for lists of clusters

Already Implemented • Working prototype • Commands: • new_query • list_clusters, list_documents • open, close (cluster) • more, back (list of clusters/documents)

Demo

Future Work • Add more functionalities (query refinement, summarization…) • Make clever use of the dialogue (not only command and control + browsing) • System can provide advice to user on search strategies (e.g. “you need to refine the query”) • User and system can negotiate to specify the user’s information need(cf Belkin: overview vs specific document)

Future Work/Discussion • Advantage of dialogue: more feedback from the user • How can dialogue improve the efficiency of low bandwidth IR? • Do we need to tailor IR techniques (e.g. clustering) for dialogue, or even design new techniques?

Vocal Access to IR • Problem: ASR introduces a lot of erroneous words in a spoken query (for an open domain, speaker independent system) • However, in an IR system: access to many text documents to help language modeling…

Vocal Access to a Newspaper Archive (Crestani 02) • Presents studies for a full voice-controlled IR system • No dialogue: user query  list of summaries • Focuses on issues of: • TTS: can user make relevance judgments when they hear document descriptions synthesized over the phone? (answer: yes) • ASR: how does IR perform with recognized queries?

Using IR Techniques to Deal with Recognition Errors • WER does have an impact on precision, although not much variation for WER in 27%-47% • Relevance feedback: use documents judged relevant by the user as query • Use prosodic stress to estimate information content of query terms • Include semantically/phonetically close terms in the query

Improving ASR (Fujii et al 02) • Fujii et al propose LM adaptation based on the IR corpus: • Offline “adaptation”: train on the whole corpus • Online adaptation: adapt on the top retrieved documents (then reperform ASR and IR) • Good results with offline trained LM (WER < 20%, AP loss of 20-30% from text IR) • No evaluation of online adaptation…

Vocal Access to IR: Discussion • Seems to work ok for some tasks • Clever use of IR techniques • BUT queries are not spontaneous nor natural (maybe) • LM for Web queries?? • What about dialogue?

(Spoken) Dialogue and Information Retrieval