Learning Text Mining

1. Learning Text Mining Walter Daelemans daelem@uia.ua.ac.be http://cnts.uia.ac.be CNTS, University of Antwerp, Belgium ILK, Tilburg University, Netherlands Language Technology BERGEN 2002

2. Outline • Using a Language Bank for Text Mining • Shallow Parsing for Text Mining • Example: Question Answering • Memory-Based Learning • Memory-Based Shallow Parsing • Tagging • Chunking • Relation-finding • Memory-Based Information Extraction

3. Text Mining • Automatic extraction of reusable information (knowledge) from text, based on linguistic features of the text • Goals: • Data mining (KDD) from unstructured and semi-structured data • (Corporate) Knowledge Management • “Intelligence” • Examples: • Email routing and filtering • Finding protein interactions in biomedical text • Matching resumes and vacancies

4. Author Recognition Document Dating Language Identification Text Categorization Information Extraction Summarization Question Answering Topic Detection and Tracking Document Clustering Terminology Extraction Ontology Extraction Document Set of Documents Structured Information + existing data Knowledge Discovery

5. Text Applications LE Components Lexical / Morphological Analysis OCR Spelling Error Correction Tagging Grammar Checking Chunking Information retrieval Syntactic Analysis Grammatical Relation Finding Document Classification Information Extraction Named Entity Recognition Summarization Word Sense Disambiguation Question Answering Semantic Analysis Ontology Extraction and Refinement Reference Resolution Dialogue Systems Discourse Analysis Machine Translation Meaning

6. Text Applications LE Components Lexical / Morphological Analysis OCR Spelling Error Correction Grammar Checking Shallow Parsing Information retrieval Document Classification Information Extraction Named Entity Recognition Summarization Word Sense Disambiguation Question Answering Semantic Analysis Ontology Extraction and Refinement Reference Resolution Dialogue Systems Discourse Analysis Machine Translation Meaning

7. Text Applications LE Components Lexical / Morphological Analysis OCR Spelling Error Correction Grammar Checking Shallow Parsing Information retrieval Document Classification Information Extraction Named Entity Recognition Summarization Word Sense Disambiguation Question Answering Semantic Analysis Ontology Extraction and Refinement Reference Resolution Dialogue Systems Discourse Analysis Machine Translation Meaning

8. Example: Shallow Parsing for Question Answering • Give answer to question (document retrieval: find documents relevant to query) • Who invented the telephone? • Alexander Graham Bell • When was the telephone invented? • 1876 (Buchholz & Daelemans, 2001)

9. QA System: Shapaqa • Parse question When was the telephone invented? • Which slots are given? • Verb invented • Object telephone • Which slots are asked? • Temporal phrase linked to verb • Document retrieval on internet with given slot keywords • Parsing of sentences with all given slots • Count most frequent entry found in asked slot (temporal phrase)

10. Shapaqa: example • When was the telephone invented? • Google: inventedAND“the telephone” • produces 835 pages • 53 parsed sentences with both slots and with a temporal phrase is through his interest in Deafness and fascination with acoustics that the telephone was inventedin 1876 , with the intent of helping Deaf and hard of hearing The telephone was invented by Alexander Graham Bell in 1876 When Alexander Graham Bell inventedthe telephonein 1876 , he hoped that these same electrical signals could

11. Shapaqa: example (2) • So when was the phone invented? • Internet answer is noisy, but robust • 17: 1876 • 3: 1874 • 2: ago • 2: later • 1: Bell • … • System was developed quickly • Precision 76% (Google 31%) • International competition (TREC): MRR 0.45

12. 4 x OSWALD * www.anusha.com/jfk.htm situation in which Oswald shot Kennedy on November 22 , 1963 . * www.mcb.ucdavis.edu/people/hemang/spooky.html Lee Harvey Oswald shot Kennedy from a warehouse and ran . * www.gallica.co.uk/monarch.htm November 1963 U.S. President Kennedy was shot by Lee Harvey Oswald . * astrospeak.indiatimes.com/mystic_corner.htm Lee Harvey Oswald shot Kennedy from a warehouse and fled . 2 x BISHOP * www.powells.com/biblio/0-200/000637901X.html The day Kennedy was shot by Jim Bishop . * www.powells.com/biblio/49200-49400/0517431009.html The day Kennedy was shot by by Jim Bishop . 1 x BULLET * www.lustlysex.com/index_m.htm President John F. Kennedy was shot by a Republican bullet . 1 x MAN * www.ncas.org/condon/text/appndx-p.htm KENNEDY ASSASSINATION Kennedy was shot by a man who was not . Who shot Kennedy ?

13. Documents Text Analysis / Shallow Parsing Analyzed Text who, what, where, when, why, how, … Text Mining Structured Data / Knowledge

14. Language Bank Classifiers Documents Text Analysis / Shallow Parsing Analyzed Text who, what, where, when, why, how, … Text Mining Structured Data / Knowledge

16. Generalisation  Abstraction Rule Induction Connectionism Inductive Logic Programming Statistics Handcrafting + abstraction … (Fill in your most hated linguist here) + generalisation - generalisation Memory-Based Learning Table Lookup - abstraction

17. MBL: Use memory traces of experiences as a basis for analogical reasoning, rather than using rules or other abstractions extracted from experience and replacing the experiences. This “rule of nearest neighbor” has considerable elementary intuitive appeal and probably corresponds to practice in many situations. For example, it is possible that much medical diagnosis is influenced by the doctor's recollection of the subsequent history of an earlier patient whose symptoms resemble in some way those of the current patient. (Fix and Hodges, 1952, p.43)

18. -etje MBL -kje Coda last syl ? Nucleus last syl

19. Memory-Based Learning • Basis: k nearest neighbor algorithm: • store all examples in memory • to classify a new instance X, look up the k examples in memory with the smallest distance D(X,Y) to X • let each nearest neighbor vote with its class • classify instance X with the class that has the most votes in the nearest neighbor set • Choices: • similarity metric • number of nearest neighbors (k) • voting weights

20. The properties of NLP tasks … • NLP tasks are mappings between linguistic representation levels that are • context-sensitive (but mostly local!) • complex (sub/ir/regularity), pockets of exceptions • Similar representations at one linguistic level correspond to similar representations at the other level • Several information sources interact in (often) unpredictable ways at the same level • Data is sparse

21. … fit the bias of MBL • The mappings can be represented as (cascades of) classification tasks (disambiguation or segmentation) • Locality is implemented through windowing over representations • Inference is based on Similarity-Based / Analogical Reasoning • Adaptive data fusion / relevance assignment is available through feature weighting • It is a non-parametric approach • Similarity-based smoothing is implicit • Regularities and subregularities / exceptions can be modeled uniformly

22. Shallow Parsing Classifiers The woman will give Mary a book

23. POS Tagging The/Det woman/NN will/MD give/VB Mary/NNP a/Det book/NN

24. Chunking [The/Det woman/NN]NP[will/MD give/VB]VP[Mary/NNP]NP[a/Det book/NN]NP

25. Relation Finding

26. Memory-Based POS Tagger Assigning morpho-syntactic categories (parts-of-speech) to words in context: Disambiguation: resolution of a combination of lexical and local contextual constraints. • Lexical representations: Frequency-sensitive ambiguity class lexicon. • Convert sentences to MBL cases by ‘windowing’. Local constraints are modeled by features of neighboring words.

27. Memory-Based POS Tagger (2) • Case base for known words. Features: • Case base for unknown words. Features: tag-2, tag-1, lexfocus, word(top100)focus, lex+1, lex+2 POS tag tag-2, tag-1, pref, cap, hyp, num, suf1, suf2, suf3, lex+1, lex+2  POS tag

28. Memory-Based POS Tagger (3) • Experimental results: (Zavrel & Daelemans, 1999)

29. Memory-Based XP Chunker Assigning non-recursive phrase brackets (Base XPs) to phrases in context: Convert NP, VP, ADJP, ADVP, PrepP, and PP brackets to classification decisions (I/O/B tags) (Ramshaw & Marcus, 1995). Features: POS -2, IOBtag-2, word -2, POS -1, IOBtag-1, word -1, POS focus, wordfocus, POS +1, word +1, POS +2, word +2,  IOB tag

30. Memory-Based XP Chunker (2) • Results (WSJ corpus) (Buchholz, Veenstra, Daelemans, 1999) • Useful for: Information Retrieval, Information Extraction, Terminology Discovery, etc. • See also CoNLL-2000 Shared task at: http://lcg-www.uia.ac.be/

31. Memory-Based GR labeling Assigning labeled Grammatical Relation links between words in a sentence: GR’s of Focus with relation to Verbs (subject, object, location, …, none) Features: Focus: prep, adv-func, word+1, word0, word-1, word-2, POS+1, POS0, POS-1, POS-2, Chunk+1, Chunk0, Chunk-1, Chunk-2. Verb: POS, word, Distance: words, VPs, comma’s  GRtype

32. Memory-Based GR labeling (2) • Results (WSJ corpus) (Buchholz, Veenstra, Daelemans, 1999) • Completes shallow parser. Useful for e.g. Question Answering, IE etc. • See demo at: http://ilk.kub.nl/

33. From POS tagging to IEClassification-Based Approach • POS tagging The/Det woman/NN will/MD give/VB Mary/NNP a/Det book/NN • NP chunking The/I-NP woman/I-NP will/I-VP give/I-VP Mary/I-NP a/B-NP book/I-NP • Relation Finding [NP-SUBJ-1 the woman ] [VP-1 will give ] [NP-I-OBJ-1 Mary] [NP-OBJ-1 a book ]] • Semantic Tagging = Information Extraction [Giver the woman][will give][Givee Mary][Given a book] • Semantic Tagging = Question Answering Who will give Mary a book? [Giver ?][will give][Givee Mary][Given a book]

34. Language Bank Classifiers Documents Text Analysis / Shallow Parsing Analyzed Text who, what, where, when, why, how, … Text Mining Structured Data / Knowledge

35. Language Bank Classifiers Documents Analyzed Text who, what, where, when, why, how, … Structured Data / Knowledge

36. IE: Experimental comparison with HMM • Systems: • TK_SemTagger: Textkernel’s Memory-Based shallow semantic tagger, using TiMBL v4.0; and • TnT (Brants, 2000), trigram HMM, smoothing by deleted interpolation. Handles unknown words by successive abstraction over suffix tree. • Classification-based Information Extraction (Zavrel & Daelemans, forthcoming) • Data set: Seminar Announcements. • Score: F1 (=2*p*r/(p+r)), Exact match, all occurrences.

37. Data set Seminar Announcement data set (Freitag, 1998): 485 documents (1-243 for training, 243-283 for validation, 283-323 testing • Extracted fields: • “speaker” • “location” • “start time” • “end time”

38. Experiments: TK_SemTagger Feature set: -4 tag, word{f>4} -3 tag, word{f>4} -2 tag, word{f>4} -1 tag, word{f>4} 0 lextag, word{f>4}, suf1, suf2, pref1, cap, allcap, initial, hyp, num, at 1 lextag, word{f>4} 2 lextag, word{f>4} 3 lextag, word{f>4} 4 lextag, word{f>4}

39. Experiments: comparison Comparison best TnT and TK_SemTagger model:

40. Conclusions • Text Mining tasks benefit from linguistic analysis (shallow parsing) • Problems ranging from shallow parsing to application-oriented tasks (Information Extraction) can be formulated as classification-based learning tasks • These classifiers can be trained on Language Banks • Memory-Based Learning seems to have the right bias for this type of task (can cope with rich feature sets and exceptions)

41. Text Mining at CNTS (http://cnts.uia.ac.be) • Development of shallow parsers, lemmatizers, named-entity recognizers and other tools • ML basic research aspects, design, adaptation to specific TM applications • English, Dutch, … • Information and Ontology Extraction • BIOMINT (?) (EU), ONTOBASIS (IWT) • Summarization and automatic subtitling • MUSA (EU), ATRANOS (IWT)