Importance of Semantic Representation: Dataless Classification

1. Importance of Semantic Representation: Dataless Classification Ming-Wei Chang Lev Ratinov Dan Roth Vivek Srikumar University of Illinois, Urbana-Champaign

2. Text Categorization Classify the following sentence: Syd Millar was the chairman of the International Rugby Board in 2003. Pick a label: Class1 vs. Class2 • Traditionally, we need annotated data to train a classifier

3. Text Categorization • Humans don’t seem to need labeled data Syd Millar was the chairman of the International Rugby Board in 2003. Pick a label: Sportsvs.Finance Label names carry a lot of information!

4. Text Categorization Do wereally always need labeled data?

5. Contributions • We can often go quite far without annotated data • … if we “know” the meaning of text • This works for text categorization • ….and is consistent across different domains

6. Outline • Semantic Representation • On-the-fly Classification • Datasets • Exploiting unlabeled data • Robustness to different domains

7. Outline • Semantic Representation • On-the-fly Classification • Datasets • Exploiting unlabeled data • Robustness to different domains

8. Semantic Representation • One common representation is the Bag of Words representation • All text is a vector in the space of words.

9. Semantic Representation • Explicit Semantic Analysis • [Gabrilovich & Markovitch, 2006, 2007] • Text is a vector in the space of concepts • Concepts are defined by Wikipedia articles

10. Monetary Policy Apple IPod ESA representation ESA representation International Monetary Fund Monetary policy Economic and Monetary Union Hong Kong Monetary Authority Monetarism Central bank IPod mini IPod photo IPod nano Apple Computer IPod shuffle ITunes Explicit Semantic Analysis: Example Wikipedia article titles

11. Semantic Representation • Two semantic representations • Bag of words • ESA

12. Outline • Semantic Representation • On-the-fly Classification • Datasets • Exploiting unlabeled data • Robustness to different domains

13. Traditional Text Categorization Labeled corpus Sports Finance Semantic space A classifier

14. Dataless Classification Labeled corpus Labels Sports Finance What can we do using just the labels?

15. But labels are text too!

16. Dataless Classification New unlabeled document Labels Sports Finance Semantic space

17. What is Dataless Classification? • Humans don’t need training for classification • Annotatedtraining data not always needed • Look for the meaning of words

18. What is Dataless Classification? • Humans don’t need training for classification • Annotated training data not always needed • Look for the meaning of words

19. On-the-fly Classification New unlabeled document Labels Sports Finance Semantic space

20. On-the-fly Classification • No training data needed • We know the meaning of label names • Pick the label that is closest in meaning to the document • Nearest neighbors

21. On-the-fly Classification New unlabeled document New labels Hockey Baseball Semantic space

22. On-the-fly Classification • No need to even know labels before hand • Compare with traditional classification • Annotated training data for each label

23. Outline • Semantic Representation • On-the-fly Classification • Datasets • Exploiting unlabeled data • Robustness to different domains

24. Dataset 1: Twenty Newsgroups • Posts to newsgroups • Newsgroups have descriptive names sci.electronics = Science Electronics rec.motorbikes = Motorbikes

25. Dataset 2: Yahoo Answers • Posts to Yahoo! Answers • Posts categorized into a two level hierarchy • 20 top level categories • Totally 280 categories at the second level Arts and Humanities, Theater Acting Sports,Rugby League

26. Experiments • 20 Newsgroups • 10 binary problems (from [Raina et al, ‘06]) Religion vs. Politics.guns Motorcycles vs. MS Windows • Yahoo! Answers • 20 binary problems Health, Diet fitness vs. Health Allergies Consumer Electronics DVRs vs. Pets Rodents

27. Results: On-the-fly classification Naïve Bayes classifier Uses annotated data, Ignores labels Nearest neighbors, Uses labels, No annotated data

28. Outline • Semantic Representation • On-the-fly Classification • Datasets • Exploiting unlabeled data • Robustness to different domains

29. Using Unlabeled Data • Knowing the data collection helps • We can learn specific biases of the dataset • Potential for semi-supervised learning

30. Bootstrapping • Each label name is a “labeled” document • One “example” in word or concept space • Train initial classifier • Same as the on-the-fly classifier • Loop: • Classify all documents with current classifier • Retrain classifier with highly confident predictions

31. Co-training • Words and concepts are two independent “views” • Each view is a teacher for the other [Blum & Mitchell ‘98]

32. Co-training • Train initial classifiers in word space and concept space • Loop • Classify documents with current classifiers • Retrain with highly confident predictions of both classifiers

33. Using unlabeled data • Three approaches • Bootstrapping with labels using Bag of Words • Bootstrapping with labels using ESA • Co-training

34. More Results Co-training using just labels does as well as supervision with 100 examples No annotated data

35. Outline • Semantic Representation • On-the-fly Classification • Datasets • Exploiting unlabeled data • Robustness to different domains

36. Domain Adaptation • Classifiers trained on one domain and tested on another • Performance usually decreases across domains

37. But the label names are the same • Label names don’t depend on the domain • Label names are robust across domains • On-the-fly classifiers are domain independent

38. Example Baseball vs. Hockey

39. Conclusion • Sometimes, label names are tell us more about a class than annotated examples • Standard learning practice of treating labels as unique identifiers loses information • The right semantic representation helps • What is the right one?