1 / 8

HTST Evaluation Notes

HTST Evaluation Notes. Outline of Stable Version of HTST. Stable version of HTST contains: HT sense knowledge base Auxiliary sub-lexicons and data extracted from HT, e.g. highly polysemous words, polyseme density etc.

patterson
Télécharger la présentation

HTST Evaluation Notes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HTST Evaluation Notes

  2. Outline of Stable Version of HTST • Stable version of HTST contains: • HT sense knowledge base • Auxiliary sub-lexicons and data extracted from HT, e.g. highly polysemous words, polyseme density etc. • Context feature (USAS tags) model data extracted from OED word sense definitions. • Main software modules • CLAWS • USAS • VARD • HT-OED based components developed in SAMUELS

  3. Evaluation • HTST is evaluated on six manually annotated test texts. • Test data: • Five test texts manually annotated by Fraser; • One EEBO test text manually annotated by Jane. • Full test data set will contain ten texts. • Evaluation criteria • General performance in terms of precision • Impact of OED contextual information • Impact of time filtering • Further evaluation on full test data is under way

  4. HTST Overall Performance Note: VARD is used for EEBO sample “1621-Newes-out-of-France”, but not used for other test data.

  5. Experiment with OED Information Note: OED information helped in most cases (modern English), but decreased precision for EEBO sample. Possible cause is that OED definitions are all written in modern English.

  6. Time Filtering on EEBO sample Published in 1621 (Main HT Codes)

  7. Time Filtering on EEBO Sample Published in 1621 (Thematic Codes)

  8. Observation • On average, about 82% precision is expected. • With proper parameter setting, thematic code tagging can reach nearly 88% on some types of texts. • Need further improvement by tuning implemented methods and introducing more reliable methods. • OED data contains noise caused by the inconsistent HT versions. If OED entries can be precisely mapped to latest HT codes in future, it should improve the tagger. • Larger reliable test data is needed for further development.

More Related