1 / 9

Relevant Sentence Detection Experiment Overview

This project involves detecting the relevance of sentences at the TREC 2004 Novelty Track, with a focus on precision, recall, and F-measure evaluation. Students will work in groups to develop systems for identifying relevant sentences based on provided topics. Training topics, relevant documents, and an evaluation program are included in the dataset for system development. Deadline for the experiment and demo is 6/21.

gad
Télécharger la présentation

Relevant Sentence Detection Experiment Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Information Retrieval and Extration 期末專題實驗 — Relevant Sentence Detection

  2. Overview • 實驗目標 • 句子層次的主題相關性偵測 • 分組 • 每組1~4人 • Deadline and Demo • 6/21

  3. Typical Ad Hoc Retrieval

  4. Relevant Sentence Detection

  5. Topics and Collection • TREC 2004 Novelty Track • 10 topics and their relevance judgments for system training and parameter tuning • “TrainingTopics.txt”in the dataset • 3-5 testing topics for demo

  6. Topic Example <top> <num>Number: N54 <title> Firestone Tire Recall <toptype> Event <desc>Description: The widespread affects of the Firestone tire recall <narr>Narrative: Opinion of the public, personal, business, or company asto the general scope of the recall (too much, not enough);as to the type of tires or vehicle that should beincluded; as well as any customer complaints or onany actions taken are relevant. Documents that brieflyreport on the recall with no enlightening details are notrelevant. <documents> APW20000808.0166 NYT20000809.0226 </top>

  7. Relevant Document Example <DOC> <DOCNO> <sdocid="APW20000808.0166"num="1"> APW20000808.0166</s> </DOCNO> <DOCTYPE> <sdocid="APW20000808.0166"num="2"> NEWS STORY</s> </DOCTYPE> <DATE_TIME> <sdocid="APW20000808.0166"num="3"> 2000-08-08 21:19</s> </DATE_TIME> <BODY> <HEADLINE> <sdocid="APW20000808.0166"num="4"> Source: Firestone To Recall Tires</s> </HEADLINE> <sdocid="APW20000808.0166"num="5"> By NEDRA PICKLER</s> <TEXT> <sdocid="APW20000808.0166"num="6"> Most of the Firestone ATX, ATX II and Wilderness AT tires are on Ford Explorers, the industry's top-selling SUV, but the recall will include tires on all brands of vehicles, the source said on condition of anonymity.</s> <sdocid="APW20000808.0166"num=“7"> The recalled tires will be replaced by other Firestone tires, the source said.</s> </TEXT> </BODY> </DOC>

  8. Evaluation • Precision, Recall and F-measure • Usage of the evaluation program (“04.eval_novelty_run.pl” in the dataset)

  9. List of Content in the Dataset • “TrainingTopics.txt”(file) • Training topics for system development • “RelevantDocsForTrainingTopics”(dir) • Relevant documents for each training topic • “04.qrels.relevant(TrainingTopics).txt” (file) • The set of relevant sentences for training topics • “04.eval_novelty_run.pl” (file) • Program for evaluation • “AdditionalDocuments(LATIMES)” (dir) • Additional (not necessary) document collection

More Related