Probase : A Knowledge Base for Text Understanding

1. Probase : A Knowledge Base for Text Understanding

2. Integrating, representing, and reasoning over human knowledge: the grand challenge of the 21stcentury.

3. Knowledge Bases • Scope: • Cross domain / Encyclopedia: (Freebase, Cyc, etc) • Vertical domain / Directory: (Entity Cube, etc) • Challenges: • completeness (e.g., a complete list of restaurants) • accuracy(e.g., address, phone, menu of each restaurant) • data integration

4. “By being able to place something into its proper place in the concept hierarchy, one can learn a considerable amount about it.” “Concepts are the glue that holds our mental world together.”

5. Yellow Pages or Knowledge Bases? • Existing Knowledge Bases are more like big Yellow Page Books • Goals: • Complete list of items • Complete information of each item • But humans do not need “complete” knowledge in order to understand the world

6. What’s wrong with the yellow pages approach? • Rich in instances, poor in concepts • Understanding = forming concepts • Ontologies always have an elegant hierarchy? • Manually crafted ontologies are for human use • Our mental world does not have an elegant hierarchy

7. Probase: “It is better to use the world as its own model.” Capture concepts (in our mental world) Quantify uncertainty (for reasoning)

8. 1 More than 2.7 million concepts automatically harnessed from 1.68 billion documents 2 Computation/Reasoning enabled by scoring: Consensus: e.g., is there a company called Apple? Typicality: e.g. how likely you think of Apple when you think about companies? Ambiguity: e.g., does the word Apple, sans any context, represent Apple the company? Similarity:e.g., how likely is an actor also a celebrity? Freshness:e.g., Pluto as a dwarf planet is a claim more fresh than Pluto as a planet. … 4 A little knowledge goes a long way after machines acquire a human touch 3 Give machines a new CPU (Commonsense Processing Unit) powered by a distributed graph engine called Trinity.

9. Probase has a big concept space Probase: Freebase:Cyc:

10. Probase has 2.7 million concepts(frequency distribution)

11. Uncertainty Probasevs. Freebase

12. Probase Internals artist painter Born Died … Movement Picasso 1881 1973 … Cubism art created by painting Year Type … Guernica … 1937 Oil on Canvas

13. Data Sources • Patterns for single statements NP such as {NP, NP, ..., (and|or)} NP such NP as {NP,}* {(or|and)} NP NP {, NP}* {,} or other NP NP {, NP}* {,} and other NP NP {,} including {NP ,}* {or | and} NP NP {,} especially {NP,}* {or|and} NP • Examples: • Good: “rich countries such as USA and Japan …” • Tough: “animals other than cats such as dogs …” • Hopeless: “At Berklee, I was playing with cats such as Jeff Berlin, Mike Stern, Bill Frisell, and Neil Stubenhaus.”

14. Examples Animals other than dogs such as cats … Eating fruits such as apple … can be good to your health. Eating disorders such as … can be bad to your health.

15. Properties • Given a class, find its properties • Candidate seed properties: • “What is the [property] of [instance]?” • “Where”, “When”, “Who” are also considered

16. Similarity between two concepts • Weighted linear combinations of • Similarity between the set of instances • Similarity between the set of attributes • (nation, country) • (celebrity, well-known politicians)

17. Beyond noun phrases • Example: the verb “hit” • Small object, Hard surface • (bullet, concrete), (ball, wall) • Natural disaster, Area • (earthquake, Seattle), (Hurricane Floyd, Florida) • Emergency, Country • (economic crisis, Mexico), (flood, Britain)

18. Quantify Uncertainty the foundation of text understanding and reasoning • Typicality P(concept | instance) P(instance | concept) P(concept | property) P(property | concept) • Similarity sim(concept1, concept2)

19. Text Mining / IE: State of the Art • Bag of words based approach: e.g., LDA • Based on multiple document statistics • Simple bag-of-words, no semantics • Supervised learning: e.g., CRF • Labeled training data required • Difficulty for out-of-sample features • Lack of semantics • What role can a knowledgebase play?

20. Shopping at Bellevue during TechFest Five of us bought 5 Kinects and posed in front of an Apple store. Apple store that sells fruits or apple store that sells iPads?

21. Step by Step Understanding

22. Explicit Semantic Analysis (ESA) • Goal: latent topics => explicit topics • Approach: mapping text to Wikipedia articles • An inverted list that record words’ occurrence in wiki articles • Given a document, we derive a distribution of wiki articles

23. Explicit Semantic Analysis (ESA) bag of words => bag of Wikipedia articles(a distribution over Wikipedia articles) A bag of Wikipedia articles is not equivalent to a clear concept in our mental world. Top 10 concepts for “Bank of America” Bank, Bank of America, Bank of America Plaza (Atlanta), Bank of America Plaza (Dallas), MBNA, VISA (credit card), Bank of America Tower New York City, NASDAQ, MasterCard, Bank of America Corporate Center

24. Short Text • Challenge: • Not enough statistics • Applications • Twitter • Query/Search Log • Anchor Text • Image/video tag • Document paraphrasing and annotation

25. Comparison of Knowledge Bases

26. When the machine sees the word ‘apple’

27. When the machine sees ‘apple’ and ‘pear’ together

28. When the machine sees ‘China’ and ‘Israel’ together

29. What China is but Israel is not?

30. What Israel is but China is not?

31. What’s the difference from a text cloud?

33. When a machine sees attributes …

34. Entity Abstraction • Given a set of entities • (Naïve Bayes Rule) • Where is a concept, and • is computed based on the concept-entity co-occurrence

35. How to Infer Concept from Attribute? (university, floridastate university, 75) (university, harvard university, 388) (university, university of california, 142) (country, china, 97346) (country, the united states , 91083) (country, india , 80351) (country, canada , 74481) • (floridastate university, website, 34) • (harvarduniversity, website, 38) • (university of california, city, 12) • (china, capital, 43) • (the united states , capital, 32) • (india, population, 35) • (canada, population, 21) • (university, website, 4568) • (university, city, 2343) • (country, capital, 4345) • (country, population, 3234) • …… • Given a set of attributes • The Naïve Bayes Rule gives • where

36. Examples • Concepts related to entities - “china” and “india” • Concepts related to attributes – “language” and “population”

37. When Type of Term is Unknown: Given a set of terms with unknown types Generative model Using Naive Bayesian rule gives: Discriminative model (Noisy-OR) And using twice Bayesian rule gives: where indicate “entity” and indicate “attribute”

38. Examples Given “china”, “india”, “language” and “population”, “emerging market” will be ranked as 1st

39. Example (Cont’d)

40. Clustering Twitter Messages • Problem 1 (unique concepts): use keywords to retrieve tweets in 3 categories: • 1. Microsoft, Yahoo, Google, IBM, Facebook • 2. cat, dog, fish, pet, bird • 3. Brazil, China, Russia, India • Problem 2 (concepts with subtle differences): use keywords to retrieve tweets in 4 categories: • 1. United states, American, Canada • 2. Malaysia, China, Singapore, India, Thailand, Korea • 3. Angola, Egypt, Sudan, Zambia, Chad, Gambia, Congo • 4. Belgium, Finland, France, Germany, Greece, Spain, Switzerland

41. Comparison Results

42. Other Possible Applications • Query Expansion • Information retrieval • Content based advertisement • Short Text Classification/Clustering • Twitter analysis/search • Text block tagging; image/video surrounding text summarization • Document Classification/Clustering/Summarization • News analysis • Out-of-sample example classification/clustering

43. © 2011 Microsoft Corporation. All rights reserved. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.