An Introduction to Natural Language Inference

1. An Introduction toNatural Language Inference Jianhao Shen 1801111357 2019/05/29

2. Outline • Introduction • Enhanced sequential inference model(ESIM) • Neural Natural Language Inference Models Enhanced withExternal Knowledge(KIM) • Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference(DMAN) • Multi-Task Deep Neural Networks for Natural Language Understanding(MT-DNN) • I Know What You Want: Semantic Learning for Text Comprehension(BERT+SRL)

3. Example (Premise)

4. Natural Language Inference(NLI) • Does premise P entails hypothesis H, they are contradict to each other, or they have no relation? • Intuitive notion of inference • One step inference, not long chains of deduction • NLI is a necessary condition for real NLU • Reasoning/inference • Global knowledge/common sense

5. Approach • Convert sentences into logic forms • Hard(Semantic parsing is still very difficult) • Data-driven • Large dataset, e.g. SNLI(2015) and MultiNLI(2017)

6. Leaderboard ESIM 88.0 ELMo ELMo GPT BERT

7. Outline • Introduction • Enhanced sequential inference model(ESIM) • Neural Natural Language Inference Models Enhanced withExternal Knowledge(KIM) • Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference(DMAN) • Multi-Task Deep Neural Networks for Natural Language Understanding(MT-DNN) • I Know What You Want: Semantic Learning for Text Comprehension(BERT+SRL)

8. ESIM • Key idea • Co-attention • Local inference • Inference composition Chen et al., Enhanced LSTM for Natural Language Inference ACL17

9. Co-attention • Soft alignment Two dogs are running through a field. There are animals outdoors.

10. Local inference • Represent each word using another sentence • Enhancement of local inference information • Element-wise product for similarity • Element-wise difference for “contradiction”

11. Inference composition • BiLSTM with enhanced local information • Compute both max and average pooling • MLP for classification

12. Tree-LSTM

13. Results

14. Ablation analysis 6 hours 40 hours

15. Outline • Introduction • Enhanced sequential inference model(ESIM) • Neural Natural Language Inference Models Enhanced withExternal Knowledge(KIM) • Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference(DMAN) • Multi-Task Deep Neural Networks for Natural Language Understanding(MT-DNN) • I Know What You Want: Semantic Learning for Text Comprehension(BERT+SRL)

16. KIM • Global knowledge/common sense help inference • P: A lady standing in a wheat field. • H: The person standing in a corn field. Chen et al., Neural Natural Language Inference Models Enhanced with External Knowledge ACL18

17. External Knowledge • Only consider lexical-level semantic knowledge: • represent relation between words wi and wj as rij • Use relations in WordNet: • Synonymy(同义词) • Antonymy(反义词) • Hypernymy(上位词) • Hyponymy(下位词) • Co-hyponyms(拥有同上位词但不同义)e.g. [dog, wolf]=1 • TransE: No improvement

18. Knowledge-Enriched Co-Attention External knowledge: Word pairs with semantic relationship may be aligned together

19. Local Inference with External Knowledge

20. Result WordNet Baseline: 85.8

21. Outline • Introduction • Enhanced sequential inference model(ESIM) • Neural Natural Language Inference Models Enhanced withExternal Knowledge(KIM) • Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference(DMAN) • Multi-Task Deep Neural Networks for Natural Language Understanding(MT-DNN) • I Know What You Want: Semantic Learning for Text Comprehension(BERT+SRL)

22. DM Prediction • Discourse Marker connects two sentences and expresses relationship between them • but, although • because, so • if, when, still, before • …… • No need to label & large data! • BookCorpus • (S1, S2, m) • 6,527,128 pairs for 8 DM

23. DMAN rp rh Pan et al., Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference ACL18

24. Result

25. Outline • Introduction • Enhanced sequential inference model(ESIM) • Neural Natural Language Inference Models Enhanced withExternal Knowledge(KIM) • Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference(DMAN) • Multi-Task Deep Neural Networks for Natural Language Understanding(MT-DNN) • I Know What You Want: Semantic Learning for Text Comprehension(BERT+SRL)

26. MT-DNN • Multi-task learning(MTL) • Not enough data in one task • Regularization • Language Model Pretraining • Universal language representation • Large (unsupervised) data Liu X, He P, Chen W, et al. Multi-Task Deep Neural Networks for Natural Language Understanding[J]. arXiv preprint arXiv:1901.11504, 2019.

27. MT-DNN

28. Results

29. Outline • Introduction • Enhanced sequential inference model(ESIM) • Neural Natural Language Inference Models Enhanced withExternal Knowledge(KIM) • Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference(DMAN) • Multi-Task Deep Neural Networks for Natural Language Understanding(MT-DNN) • I Know What You Want: Semantic Learning for Text Comprehension(BERT+SRL)

30. Semantic Role Labeling • SRL • Who did what to whom, when where and why • Shallow semantic • NLI/MRC • Text comprehension and inference • Deep semantic

31. SRL Framework

32. SRL model • spaCy: POS tags • BiLSTM: BIO encoding

33. Results

34. Summary • ESIM • Co-attention • Local inference • Inference composition • Global Knowledge • WordNet • Discourse Marker • No need to label & large data • Multi-task + LM Pretraining • Semantic role labeling

35. Thanks~Q&A