1 / 14

Adaptive Topic Tracking at Maryland

This presentation examines the results of adaptive and non-adaptive topic tracking systems based on TDT-2004 research conducted at the University of Maryland and the NSA. We analyze system designs, the impact of normalization on performance, and key findings regarding incomplete judgments. Through our evaluation, we found that normalization issues affected results significantly, and we propose next steps for improvement, including further exploration of normalization techniques and refining training data. This research aims to enhance topic tracking effectiveness for future applications.

michelle-briggs
Télécharger la présentation

Adaptive Topic Tracking at Maryland

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Adaptive Topic Tracking at Maryland Tamer Elsayed, Douglas W. Oard, David Doermann University of Maryland, College Park Gary Kuhn National Security Agency TDT-2004

  2. Outline • Results • System design • Interpreting the results • Next steps

  3. Cost=0.6507 Non-Adaptive Topic Tracking Bottom left is better No score normalization

  4. Cost=0.2438 Adaptive Topic Tracking No score normalization, unjudged treated as firmly off-topic

  5. Cost=0.3789 Adaptive Topic Tracking One-pass score normalization, unjudged treated as firmly off-topic

  6. Non-Adaptive System Design TDT-5 Training Epoch Evaluation Epoch Compute log-odds ngram weights Compute story scores

  7. Log-Odds Term Weights

  8. Computing Story Scores

  9. Non-Adaptive System Design TDT-5 Training Epoch Evaluation Epoch Compute log-odds ngram weights Compute story scores

  10. Compute Normalization factor Normalize Story scores Adaptive System Design TDT-4 TDT-5 Extended Training Epoch Training Epoch Evaluation Epoch Compute log-odds ngram weights Compute story scores

  11. Lack of normalization probably hurt! What can we say about the effect of incomplete judgments? Interpreting Non-Adaptive Results

  12. Normalization hurt! One-pass design is the problem DET has limitations Changing the threshold changes our topic model! Threshold selection is now a critical path item How does judgment density affect the results? Interpreting Adaptive Results Not normalized Normalized

  13. Next Steps • Further explore normalization • Implement continuous renormalization • Tune parameters on devtest data • Decide between TDT-5 and TDT-4 • Is incomplete judging harmful? • Define richer training sets • Explicit queries • Many known on-topic/off-topic training stories • Models of (imperfect) behavioral feedback

  14. Our Favorite Quote of the Day • “It takes time to get the implementation correct” [Yiming] • We had 30 days from project initiation to non-adaptive submission

More Related