Automatic Spoken Document Processing for Retrieval and Browsing

1. Automatic Spoken Document Processing for Retrieval and Browsing Zahra Ahmadi

2. Outline • Motivation • Typical speech retrieval system • Works done • Dealing with OOV • Improvements

3. Motivation • Ever-increasing computing power and connectivity bandwidth, together with falling storage costs resulting in an overwhelming amount of data of various types • Information search and retrieval is a key application area • Less attention to speech search • As data availability increases, lack of adequate technology for processing spoken documents becomes the limiting factor to large-scale access to spoken content • Automatic approaches for indexing and searching spoken document collections are very desirable

4. Typical Speech Retrieval System • Two primary processing stage: • Offline process of audio content to generate index • Query searches via interface and system’s retrieval based on indexes • ASR is the core component of speech retrieval system

5. SDR Challenges • Primary difficulties due to limitations of ASR technology: • Highly spontaneous, unprepared speech • Topic-specific or person-specific vocabulary & language usage • Unknown content and topics potentially lacking support in general language model • Wide variety of accents and speaking styles • OOVs in queries • Infrequent query terms, which are most useful for retrieval

6. Prominent Approaches • Many of prominent research efforts: SDR-TREC in 1999-2000 • Significant recent contributions on wide variety of speech sources: • SpeechBot: audio from public web sites • SCANMail: voice mail • Oral history interviews • SpeechFind: National Gallery of the Spoken Word (NGSW) consisting of speeches, news broadcasts, and recordings that are of significant historical content

7. TREC-SDR: A Success Story • About 550 hours of broadcast news • Segmented manually into 21,574 stories of 250 words on average • Evaluation of ASR systems tuned to broadcast news domain: 15-20% WER • Preexisting approximate manual transcriptions had WER of 14.5% for video and 7.5% for radio broadcasts • Accuracy evaluation: by human assessors search queries • Retrieval performance was flat with respect to ASR WER (1-best) variations in the range of 15-30% (robust to recognition errors) • No severe degradation in retrieval performance when evaluating with ASR outputs in comparison with approximate manual transcriptions

8. TREC-SDR Robustness Results

9. Shortcomings of TREC-SDR • Speech recognizers tuned heavily for domain: • Lead to very good ASR performance • Unrealistic to expect 10–15% WER especially when decoding speech is mismatched to the training data • Common to observe WER of 30-50% • Very low OOV rates: • Typically below 1% • Query-side OOV (Q-OOV) was very low as well By Q-OOV rate close to 15%, severe degradation in MAP performance (50% relative, from 44 to 22) occurs

10. Dealing with OOV Query Words • Most common: represent both query and spoken document using subword units: • Linguistically: • Phone: completely solves OOV problem, low performance • Syllable: stable acoustically, poor language model • Morpheme: hard to distinguish acoustically • Stem-ending: acceptable OOV, distinguishable segment (agglutinative lang.) • Data driven: • Multigram: non-overlapping, variable-length, phone subsequences with some predefined maximum length • Particle: found greedy to max. leave-one-out likelihood of bigram LM • Morph: based on minimum description length

11. Dealing with OOV Query Words (cont) • Advocates tighter integration of ASR and IR: • Index phone n-grams appearing in ASR N-best lists • Focused on broadcast news thus benefiting from good ASR performance • Combination of word and subword level indexing: • word-level indexing and querying is still more accurate • abundance of word-spotting false-positives in subword retrieval • somewhat masked by the MAP measure

12. Effects of Using Different Methods

13. Dealing with OOV Query Words (cont) • Building inverted index from ASR lattice: • Storing full connectivity information in lattice • Retrieval is performed by looking up strings of units • Allows for exact calculation of n-gram expectedcounts but more general proximity information is hard to calculate • Query expansion: • Expand to similar in-vocabulary phrases • Phone confusion matrix: acoustic confusion between words • Stemming • Semantic similarity Use of more than just one-best information (N-best lists or lattices) significantly improves retrieval accuracy

14. Long Spoken Communications • Important to locate the relevant portion • Achievable by segmenting documents into topics and locating topics • Spoken Utterance Retrieval (SUR): where segments are short or when consist of short utterances • SUR goal is to find all utterances containing query • Applications: browsing broadcast news, telephone conversations, teleconferences, and lectures • NIST STD 2006 Evaluation : • Locating exact occurrence of query in large heterogeneous speech archives • Notable technique with significant improvements: setting detection thresholds in a term-specific fashion to maximize ATWV metric

15. Spoken Document Understanding & Organization Automatically analyzing subject topics of segmented paragraphs, clustering with topic labels, organizing hierarchical presentation Automatically generating title for each short paragraph Automatically extracting key information of events in segmented short paragraphs Automatically generating summery for each segmented short paragraphs Automatically segmented into short paragraphs with some central concept Keywords in spoken document to understand subject matters

16. Finally… • Use of audio content and text metadata jointly can improve retrieval performance • Conjunction of subword and word-based methods improves performance • Need to universal ASR which controls variance in WER across narrow domains as SDR poses new challenges for the core ASR • Cross-Language SDR: Assumes queries and target spoken documents are not in the same language • Bilingual performance was lower than English monolingual run • However, the degree of degraded performance was shown to depend on the translation resources used. • Extension of TREC collections by manually translating short topics into five European languages: Dutch, French, German, Italian, Spanish

18. References • C. Chelba, T.J. Hazen, M. Saraclar. “Retrieval and Browsing of Spoken Content”. IEEE Signal Processing Magazine, May 2008. • L. Lee, B. Chen. “Spoken Document Understanding and Organization”. IEEE Signal Processing Magazine, September 2005. • J. Garofolo, G. Auzanne, and E. Voorhees. “The TREC Spoken Document Retrieval Track: A Success Story”. Proc. Recherche d’Informations Assiste par Ordinateur: Content Based Multimedia Information Access Conf., 2000. • L. Begeja, D. Gibbon, et. Al. “A System for Searching and Browsing Spoken Communications”. 2004. • S. Parlak, M. saraclar. “Spoken Term Detection for Turkish Broadcast NEWS”. ICASSP 2008. • N. Bertoldi, M. Federico. “Cross-Language Spoken Document Retrieval on the TREC SDR Collection”. Springer, pp.476-481, 2003. • C. Chelba, T.J. Hazen. “Automatic Spoken Document Processing for Retrieval and Browsing”. Tutorial slides, NAACL 2006. • …