1 / 55

RTE-7@TAC2010 The Seventh Recognizing Textual Entailment Challenge

RTE-7@TAC2010 The Seventh Recognizing Textual Entailment Challenge. Luisa Bentivogli (coordinator, CELCT & FBK-irst) Danilo Giampiccolo (coordinator, CELCT) Hoa Trang Dang (NIST) Ido Dagan (Bar Ilan University) Peter Clark (Vulcan Inc.). Outline. The RTE Challenge

kiril
Télécharger la présentation

RTE-7@TAC2010 The Seventh Recognizing Textual Entailment Challenge

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. RTE-7@TAC2010The Seventh Recognizing Textual Entailment Challenge Luisa Bentivogli(coordinator, CELCT & FBK-irst) Danilo Giampiccolo (coordinator, CELCT) Hoa Trang Dang (NIST) Ido Dagan (Bar Ilan University) Peter Clark (Vulcan Inc.)

  2. Outline • The RTE Challenge • RTE-7 Main Task: RTE within a Corpus • RTE-7 Novelty Detection Subtask • Knowledge Resources and Tools for RTE • RTE-7 KBP Validation Task • Conclusion and Future Perspectives RTE-7@TAC2011

  3. Outline • The RTE Challenge • RTE-7 Main Task: RTE within a Corpus • RTE-7 Novelty Detection Subtask • Knowledge Resources and Tools for RTE • RTE-7 KBP Validation Task • Conclusion and Future Perspectives RTE-7@TAC2011

  4. Textual entailment isadirectional relation between two text fragments: the entailing text, called T(ext) the entailed text, called H(ypothesis) Textual Entailment T entails H if, typically, a human reading T would infer that H is most likely true RTE-7@TAC2011

  5. Examples • YES T:The Christian Science Monitor named a US journalist kidnapped in Iraq as freelancer Jill Carroll. H:Jill Carroll was abducted in Iraq. • NO T:The Christian Science Monitor named a US journalist kidnapped in Iraq as freelancer Jill Carroll. H: Jill Carroll is the daughter of Mary Beth Carroll. RTE-7@TAC2011

  6. The RTE-7 Challenge Replicates the same tasks as in RTE-6 to allow participants to address the novelties introduced for the first time in RTE-6: • Main Task: TextualEntailmentwithin a Corpus (Piloted in RTE-5 - Summarizationsetting) • Novelty Detection Subtask (based on the Main Task) • KBP Validation Task (Knowledge Base Population setting) • Exploratory effort on resource evaluation extended to tools RTE-7@TAC2011

  7. RTE-7 Participants • Number of participants: 13 • RTE-1: 18, RTE-2: 23, RTE-3: 26, RTE-4: 26, RTE-5: 21, RTE-6: 18 • Provenance • ASIA: 8 • EUROPE: 5 • Participants per task • Main Task: 13 (33 runs) • Novelty Detection Subtask: 5 (13 runs) • KBP Validation Pilot Task: 2 (8 runs) RTE-7@TAC2011

  8. Outline • The RTE Challenge • RTE-7 Main Task: RTE within a Corpus • RTE-7 Novelty Detection Subtask • Knowledge Resources and Tools for RTE • RTE-7 KBP Validation Task • Conclusion and Future Perspectives RTE-7@TAC2011

  9. RTE-7 Main Task Description • Given • a corpus • a hypothesis H • a set of "candidate" entailing sentences for that H retrieved by Lucene from the corpus • RTE systems are required • to identify all the sentences among the candidate sentences that entail a given Hypothesis RTE-7@TAC2011

  10. RTE-7 Main Task Example Topic 918: Betty Friedan H380 :Betty Friedan is the author of "The Feminine Mystique." H391 : "The Feminine Mystique" was published in 1963. H401 : In 1962, Judy Mott was laid off from her job with Sears. Hs SET H380: Betty Friedan is the author of "The Feminine Mystique" Document 1 Document 2 Document 3 S1: Betty Friedan, a founder of the modern feminist movement in the United States, died here Saturday of congestive heart failure, feminist leaders announced. S2: She was 85. S3: Friedan achieved prominence in l963 with the publication of her book "The Feminine Mystique," which detailed the lives of American women who were expected to find fulfillment through the achievements of their husbands and children. S4: The book sparked a movement for a re-evaluation of women's role in American society and is credited with laying the foundation of modern feminism. S5: She was a founder of the National Organization for Women and a leading advocate of the Equal Rights Amendment, a proposed amendment to the US constitution banning sex-based discrimination, women's rights activists said. S6: "The movement that Friedan's energy sparked continues to grow, and is bigger today than she could ever have dreamed … … S1: Betty Friedan, the visionary, combative feminist who launched a social revolution with her provocative 1963 book, "The Feminine Mystique," died Saturday, which was her 85th birthday. S2: Friedan died of congestive heart failure at her home in Washington, D.C., according to Emily Bazelon, a cousin who was speaking for the family. S3: She said Friedan had been in failing health for some time. S4: Her best-selling book identified "the problem that has no name," the unhappiness of post-World War II American women unfulfilled by traditional notions of female domesticity. S5:. Melding sociology and humanistic psychology, the book became the cornerstone of one of the last century's most profound movements, unleashing the first full flowering of American feminism since the 1800s. S6: It gave Friedan, an obscure suburban New York housewife and freelance writer, the mantle to... … S26: What is perhaps most surprising, though, is not that feminists like Hirshman believe homemaking is second-class drudgery, but that so many people still get worked up over the issue. S27: After all, feminist thinkers have been proclaiming the need to free women from the bondage of housework for a long time.. S28: It is, as Hirshman freely acknowledges, precisely what Friedan argued in "The Feminine Mystique," first published more than 40 years ago. S29 "The only kind of work which permits an able woman to realize her abilities fully," Friedan wrote, "is the kind that was forbidden by the feminine mystique, the lifelong commitment to an art or science, to politics or profession.". S30:Not homemaking, not motherhood. S31: In an interview, Hirshman said that in the course of researching a book, she began to wonder when feminism switched from offering a clear blueprint for liberation to choosing from Column A and Column B. … RTE-7@TAC2011

  11. RTE-7 Main Task Example Topic 918: Betty Friedan H380 :Betty Friedan is the author of "The Feminine Mystique." H391 : "The Feminine Mystique" was published in 1963. H401 : In 1962, Judy Mott was laid off from her job with Sears. Hs SET H380: Betty Friedan is the author of "The Feminine Mystique" Document 1 Document 2 Document 3 S1: Betty Friedan, a founder of the modern feminist movement in the United States, died here Saturday of congestive heart failure, feminist leaders announced. S2: She was 85. S3: Friedan achieved prominence in l963 with the publication of her book "The Feminine Mystique," which detailed the lives of American women who were expected to find fulfillment through the achievements of their husbands and children. S4: The book sparked a movement for a re-evaluation of women's role in American society and is credited with laying the foundation of modern feminism. S5: She was a founder of the National Organization for Women and a leading advocate of the Equal Rights Amendment, a proposed amendment to the US constitution banning sex-based discrimination, women's rights activists said. S6: "The movement that Friedan's energy sparked continues to grow, and is bigger today than she could ever have dreamed … … S1: Betty Friedan, the visionary, combative feminist who launched a social revolution with her provocative 1963 book, "The Feminine Mystique," died Saturday, which was her 85th birthday. S2: Friedan died of congestive heart failure at her home in Washington, D.C., according to Emily Bazelon, a cousin who was speaking for the family. S3: She said Friedan had been in failing health for some time. S4: Her best-selling book identified "the problem that has no name," the unhappiness of post-World War II American women unfulfilled by traditional notions of female domesticity. S5:. Melding sociology and humanistic psychology, the book became the cornerstone of one of the last century's most profound movements, unleashing the first full flowering of American feminism since the 1800s. S6: It gave Friedan, an obscure suburban New York housewife and freelance writer, the mantle to... … S26: What is perhaps most surprising, though, is not that feminists like Hirshman believe homemaking is second-class drudgery, but that so many people still get worked up over the issue. S27: After all, feminist thinkers have been proclaiming the need to free women from the bondage of housework for a long time.. S28: It is, as Hirshman freely acknowledges, precisely what Friedan argued in "The Feminine Mystique," first published more than 40 years ago. S29 "The only kind of work which permits an able woman to realize her abilities fully," Friedan wrote, "is the kind that was forbidden by the feminine mystique, the lifelong commitment to an art or science, to politics or profession.". S30:Not homemaking, not motherhood. S31: In an interview, Hirshman said that in the course of researching a book, she began to wonder when feminism switched from offering a clear blueprint for liberation to choosing from Column A and Column B. … RTE-7@TAC2011

  12. RTE-7 Main Task Example Topic 918: Betty Friedan H380 :Betty Friedan is the author of "The Feminine Mystique." H391 : "The Feminine Mystique" was published in 1963. H401 : In 1962, Judy Mott was laid off from her job with Sears. Hs SET H380: Betty Friedan is the author of "The Feminine Mystique" Document 1 Document 2 Document 3 S1: Betty Friedan, a founder of the modern feminist movement in the United States, died here Saturday of congestive heart failure, feminist leaders announced. S2: She was 85. S3: Friedan achieved prominence in l963 with the publication of her book "The Feminine Mystique," which detailed the lives of American women who were expected to find fulfillment through the achievements of their husbands and children. S4: The book sparked a movement for a re-evaluation of women's role in American society and is credited with laying the foundation of modern feminism. S5: She was a founder of the National Organization for Women and a leading advocate of the Equal Rights Amendment, a proposed amendment to the US constitution banning sex-based discrimination, women's rights activists said. S6: "The movement that Friedan's energy sparked continues to grow, and is bigger today than she could ever have dreamed … … S1: Betty Friedan, the visionary, combative feminist who launched a social revolution with her provocative 1963 book, "The Feminine Mystique," died Saturday, which was her 85th birthday. S2: Friedan died of congestive heart failure at her home in Washington, D.C., according to Emily Bazelon, a cousin who was speaking for the family. S3: She said Friedan had been in failing health for some time. S4: Her best-selling book identified "the problem that has no name," the unhappiness of post-World War II American women unfulfilled by traditional notions of female domesticity. S5:. Melding sociology and humanistic psychology, the book became the cornerstone of one of the last century's most profound movements, unleashing the first full flowering of American feminism since the 1800s. S6: It gave Friedan, an obscure suburban New York housewife and freelance writer, the mantle to... … S26: What is perhaps most surprising, though, is not that feminists like Hirshman believe homemaking is second-class drudgery, but that so many people still get worked up over the issue. S27: After all, feminist thinkers have been proclaiming the need to free women from the bondage of housework for a long time.. S28: It is, as Hirshman freely acknowledges, precisely what Friedan argued in "The Feminine Mystique," first published more than 40 years ago. S29 "The only kind of work which permits an able woman to realize her abilities fully," Friedan wrote, "is the kind that was forbidden by the feminine mystique, the lifelong commitment to an art or science, to politics or profession.". S30:Not homemaking, notmotherhood. S31: In an interview, Hirshman said that in the course of researching a book, she began to wonder when feminism switched from offering a clear blueprint for liberation to choosing from Column A and Column B. … S1: Betty Friedan, the visionary, combative feminist who launched a social revolution with her provocative 1963 book, "The Feminine Mystique," died … S3: Friedan achieved prominence in l963 with the publication of her book "The Feminine Mystique," which detailed the lives of American women ... S28: It is, as Hirshman freely acknowledges, precisely what Friedan argued in her book "The Feminine Mystique," first published... RTE-7@TAC2011

  13. RTE-7 Main Data Set (1/2) Initial Summary Cluster A Time Update Summary TAC 2008 and 2009 SUM Update scenario For each topic: RTE-7@TAC2011 Cluster B

  14. RTE-7 Main Data Set (2/2) Topic 918: Betty Friedan H380: Betty Friedan is the author of "The Feminine Mystique." H381: Betty Friedan died on February 4, 2006. H382: Betty Friedan died at 85. H397: In The Guardian, Germaine Greer took critical measure of Betty Friedan. Hs SET Document 1 Document 2 Document 3 S1: Betty Friedan, a founder of the modern feminist movement in the United States, died here Saturday of congestive heart failure, feminist leaders announced. S2: She was 85. S3: Friedan achieved prominence in l963 with the publication of her book "The Feminine Mystique," which detailed the lives of American women who were expected to find fulfillment through the achievements of their husbands and children. S4: The book sparked a movement for a re-evaluation of women's role in American society and is credited with laying the foundation of modern feminism. S5: She was a founder of the National Organization for Women and a leading advocate of the Equal Rights Amendment, a proposed amendment to the US constitution banning sex-based discrimination, women's rights activists said. S6: "The movement that Friedan's energy sparked continues to grow, and is bigger today than she could ever have dreamed … … S1: Betty Friedan, the visionary, combative feminist who launched a social revolution with her provocative 1963 book, "The Feminine Mystique," died Saturday, which was her 85th birthday. S2: Friedan died of congestive heart failure at her home in Washington, D.C., according to Emily Bazelon, a cousin who was speaking for the family. S3: She said Friedan had been in failing health for some time. S4: Her best-selling book identified "the problem that has no name," the unhappiness of post-World War II American women unfulfilled by traditional notions of female domesticity. S5:. Melding sociology and humanistic psychology, the book became the cornerstone of one of the last century's most profound movements, unleashing the first full flowering of American feminism since the 1800s. S6: It gave Friedan, an obscure suburban New York housewife and freelance writer, the mantle to... … S26: What is perhaps most surprising, though, is not that feminists like Hirshman believe homemaking is second-class drudgery, but that so many people still get worked up over the issue. S27: After all, feminist thinkers have been proclaiming the need to free women from the bondage of housework for a long time.. S28: It is, as Hirshman freely acknowledges, precisely what Friedan argued in "The Feminine Mystique," first published more than 40 years ago. S29 "The only kind of work which permits an able woman to realize her abilities fully," Friedan wrote, "is the kind that was forbidden by the feminine mystique, the lifelong commitment to an art or science, to politics or profession.". S30:Not homemaking, not motherhood. S31: In an interview, Hirshman said that in the course of researching a book, she began to wonder when feminism switched from offering a clear blueprint for liberation to choosing from Column A and Column B. … • 20-40 standalone sentences: • based on the “B” summary sentences of • the 3 best scoring SUM systems • - based directly on Cluster “A” sentences • Automatic summary sentence: • In The Guardian, Germaine Greer took critical measure of a fellow feminist, Betty Friedan, theauthor of “The Feminine Mystique” who died on Feb. 4at 85. Cluster A RTE-7@TAC2011

  15. RTE-7 Main Data Set (2/2) • Up to 100 “candidate” entailing sentences • - Information Retrievalfilteringphase: • - The H is the query • - The corpus sentences are “the documents” tobe • retrievedfor the query • - the 100 top-ranked sentences are selected as • candidates (80% of all the entailing sentences in the corpus) • - LUCENE text searchengine (v. 2.9.1): • - StandardAnalyzer, Boolean “OR” query, • Default Lucene ranking Topic 918: Betty Friedan H380 :Betty Friedan is the author of "The Feminine Mystique." H391 : "The Feminine Mystique" was published in 1963. H401 : In 1962, Judy Mott was laid off from her job with Sears. Hs SET H380: Betty Friedan is the author of "The Feminine Mystique" Document 1 Document 2 Document 3 S1: Betty Friedan, a founder of the modern feminist movement in the United States, died here Saturday of congestive heart failure, feminist leaders announced. S2: She was 85. S3: Friedan achieved prominence in l963 with the publication of her book "The Feminine Mystique," which detailed the lives of American women who were expected to find fulfillment through the achievements of their husbands and children. S4: The book sparked a movement for a re-evaluation of women's role in American society and is credited with laying the foundation of modern feminism. S5: She was a founder of the National Organization for Women and a leading advocate of the Equal Rights Amendment, a proposed amendment to the US constitution banning sex-based discrimination, women's rights activists said. S6: "The movement that Friedan's energy sparked continues to grow, and is bigger today than she could ever have dreamed … … S1: Betty Friedan, the visionary, combative feminist who launched a social revolution with her provocative 1963 book, "The Feminine Mystique," died Saturday, which was her 85th birthday. S2: Friedan died of congestive heart failure at her home in Washington, D.C., according to Emily Bazelon, a cousin who was speaking for the family. S3: She said Friedan had been in failing health for some time. S4: Her best-selling book identified "the problem that has no name," the unhappiness of post-World War II American women unfulfilled by traditional notions of female domesticity. S5:. Melding sociology and humanistic psychology, the book became the cornerstone of one of the last century's most profound movements, unleashing the first full flowering of American feminism since the 1800s. S6: It gave Friedan, an obscure suburban New York housewife and freelance writer, the mantle to... … S26: What is perhaps most surprising, though, is not that feminists like Hirshman believe homemaking is second-class drudgery, but that so many people still get worked up over the issue. S27: After all, feminist thinkers have been proclaiming the need to free women from the bondage of housework for a long time.. S28: It is, as Hirshman freely acknowledges, precisely what Friedan argued in "The Feminine Mystique," first published more than 40 years ago. S29 "The only kind of work which permits an able woman to realize her abilities fully," Friedan wrote, "is the kind that was forbidden by the feminine mystique, the lifelong commitment to an art or science, to politics or profession.". S30:Not homemaking, not motherhood. S31: In an interview, Hirshman said that in the course of researching a book, she began to wonder when feminism switched from offering a clear blueprint for liberation to choosing from Column A and Column B. … RTE-7@TAC2011

  16. Data Set Composition 3 annotations for the whole data set IAA (Kappa): 98.35% (Dev), 98.51% (Test) RTE-7@TAC2011

  17. Main Task Evaluation 13 participants (33 runs) • Evaluation measures: • Precision, Recall, F-measure (micro-averaged) • IR Baselines: RTE-7@TAC2011

  18. Best Results RTE-7@TAC2011

  19. Results: F-measure statistics RTE-7@TAC2011

  20. Results: F-measure statistics RTE-7@TAC2011

  21. Outline • The RTE Challenge • RTE-7 Main Task: RTE within a Corpus • RTE-7 Novelty Detection Subtask • Knowledge Resources and Tools for RTE • RTE-7 KBP Validation Task • Conclusion and Future Perspectives RTE-7@TAC2011

  22. RTE-7 Novelty Detection Subtask Goals: • Specifically address the needs of the SUM Update Task, where it is necessary to distinguish between novel and non novel information • RTE engines could help summarization systems to filter out non-novel sencences from their summaries RTE-7@TAC2011

  23. RTE-7 Novelty Detection Subtask Task: Judgeif the information contained in each H (from Cluster B) isnovelwithrespectto the information contained in the set of (Cluster A) candidate entailingsentences • If a given H: • has entailing sentences = information is NOT novel • has not entailing sentences = information is novel RTE-7@TAC2011

  24. RTE-7 Novelty Detection Subtask Based on the Main Task: • Uses only the Hs taken from the automatic summaries • Same output format/annotation • the novelty detection decision is derived automatically from the number of entailing sentences for each H Differences: • Systems are specifically tuned for novelty detection • Specific scoring metrics designed for assessing novelty detection RTE-7@TAC2011

  25. Data Set Composition IAA (Kappa): 98.21% (Dev), 98.06% (Test) RTE-7@TAC2011

  26. Evaluation Measures 5 participants (13 runs) • Primary score: Novelty Detection evaluation • Micro Averaged Precision, recall and F-measure computed on the binary novel/non-novel decision • derived automatically from the number of entailing sentences provided by the systems • Secondary score: Justification evaluation • measures the quality of the justifications provided for non-novel Hs • Micro-averaged Precision, Recall and F-measure on the set of all the sentences extracted as entailing the Hs RTE-7@TAC2011

  27. Best Results – Primary Score RTE-7@TAC2011

  28. Best Results – Secondary Score RTE-7@TAC2011

  29. Results: F-measure statistics RTE-7@TAC2011

  30. Results: F-measure statistics RTE-7@TAC2011

  31. Outline • The RTE Challenge • RTE-7 Main Task: RTE within a Corpus • RTE-7 Novelty Detection Subtask • Knowledge Resources and Tools for RTE • RTE-7 KBP Validation Task • Conclusion and Future Perspectives RTE-7@TAC2011

  32. Knowledge Resources and Tools for RTE An exploratory effort aimed at studying the relevance of knowledge resources and tools in recognizing TE • Ablation Tests for all knowledge resources and tools used in Main Task runs: • remove one module at a time from a system, and re-run the system on the test set with the other modules, except the one tested • Removeonlyknowledgeresources or tools • Removeoneresource or tool at a time RTE-7@TAC2011

  33. Ablation Tests • 31 ablation tests submitted (by 10 teams) • 7 tests did not specifically address knowledge resources or tools • 3 tests had a combination of different resources/components removed • 21 ablation tests conformant to the requirements • 16 tests for 7 different resources • 5 tests for 2 different tools RTE-7@TAC2011

  34. Ablation Tests - Resources RTE-7@TAC2011

  35. Ablation Tests - Resources RTE-7@TAC2011

  36. Ablation Tests - Tools RTE-7@TAC2011

  37. Ablation Tests - Tools RTE-7@TAC2011

  38. Remarks on the initiative • WRT RTE-5 and RTE-6: • Resources: trends confirmed over the years • Tools: RTE-6 trends not confirmed • Lesson learned • Ablation test results may provide an indication of the actual contribution of a component to the performance a specific system • BUT the value of a resource is very much dependent on how that resource is used and how it integrates with the rest of the system • Need for a deeper comprehension of the usage of the resources and tools RTE-6@TAC2010

  39. Outline • The RTE Challenge • RTE-7 Main Task: RTE within a Corpus • RTE-7 Novelty Detection Subtask • Knowledge Resources and Tools for RTE • RTE-7 KBP Validation Task • Conclusion and Future Perspectives RTE-7@TAC2011

  40. The RTE-7 KBP Validation Task Motivations: • analyze the potential utility of RTE systems in another real NLP application scenario, i.e. the Knowledge Base Population Slot Filling task • use Textual Entailment techniques to validate the output of an NLP system (similar to the AVE experiment in QA) RTE-7@TAC2011

  41. The KBP Slot Filling Task Given an entity in a knowledge base and an attribute (slot) for that entity: • find in a large corpus the correct value (filler) for that attribute • return the extracted information together with a corpus document supporting it as a correct slot filler RTE-7@TAC2011

  42. The RTE-7 KBP Validation Task Initial assumption: an extracted slot filler is correct if and only if the supporting document entails a hypothesis summarizing the slot filler Task : determine whether a candidate slot filler is supported in the associated document using entailment techniques. RTE-7@TAC2011

  43. Data Set Creation • 1 RTE evaluationpair, where: • T is the entiredocumentsupporting the slot filler • H is a set ofsynonymoussentences, representingdifferentrealizations of the slot filler Each slot filler returned by KBP systems RTE-7@TAC2011

  44. Data Set Creation: example KBP SYSTEM INPUT KBP SYSTEM OUTPUT Target Entity: Chris Simcox Slot: Residences Document collection Slot Filler: “Tucson, Ariz.” Supporting Document: NYT_ENG_20050919.0130.LDC2007T07 RTE EVALUATION PAIR T: NYT_ENG_20050919.0130.LDC2007T07 RTE-7@TAC2011

  45. Hypotheses Creation Manuallycreatedtemplates Hs Attribute:origin Target entity:person Slot filler:Canadian Target entity:Chris Simcox H1 Chris Simcoxorigins are in Canadian H2 Chris Simcoxcomes from Canadian H3 Chris Simcox is from Canadian H4 Chris Simcoxorigins are Canadian H5 Chris Simcoxhas Canadian origin H6 Chris Simcoxis of Canadian origin Templ1: X’s origins are inY Templ2: Xcomes fromY Templ3: Xis fromY Templ4: Xorigins areY Templ5: X hasYorigins Templ6: X is ofYorigin RTE-7@TAC2011

  46. Gold Standard Creation KBP JUDGMENTS ENTAILMENT VALUES (4-valued) (2-valued) Correct YES Redundant YES Wrong NO Inexact (notincluded) KBP assessments (automatically) RTE gold standard annotations RTE-7@TAC2011

  47. Distinguishing Features • RTE evaluation pair • T is an entire document • H is a set of synonymous sentences, possibly ungrammatical • (Semi-)automatic generation • Data Set • from KBP outputs • Gold Standard • from KBP output assessments RTE-7@TAC2011

  48. Data Set Composition Removed pair types: GPE; “inexact”; “NO_RESPONSE”; duplicates; speech transcriptions; “other_family” slot ; web documents RTE-7@TAC2011

  49. Evaluation 2 types of submissions: • genericsystems (no adaptation) • tailoredsystems (adaptedforspecificslots) Participants : 2 Submittedruns: 8 • 5 generic • 3 tailored Evaluationmeasures: Micro-AveragedPrecision, Recall, F-measure RTE-7@TAC2011

  50. Pilot Task Baseline Baseline: AllTsclassifiedasentailing the corresponding H Thisbaseline: • reflects the cumulative performance of all KBP Slot FillingSystems • indicates the percentage of entailingpairs in the Test Set RTE-7@TAC2011

More Related