1 / 29

Teamware: A Collaborative, Web-based Annotation Environment

Teamware: A Collaborative, Web-based Annotation Environment. Kalina Bontcheva, Milan Agatonovic University of Sheffield. Hands-on Preparation. Go to the FIG’09 Wiki http://gate.ac.uk/wiki/Wiki.jsp?page=FIG09 Under Resources, Teamware lecture Click on link to the Teamware install

morty
Télécharger la présentation

Teamware: A Collaborative, Web-based Annotation Environment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Teamware: A Collaborative,Web-based Annotation Environment Kalina Bontcheva, Milan AgatonovicUniversity of Sheffield

  2. Hands-on Preparation Go to the FIG’09 Wiki http://gate.ac.uk/wiki/Wiki.jsp?page=FIG09 Under Resources, Teamware lecture Click on link to the Teamware install Login using you user name (from your reg.pack): <cics-account-id>-annotator Click on the link “Annotation Editor” to download and prepare the software for our first hands on When it opens, leave it as is, till we need it GATE Summer School - July 27-31, 2009 2

  3. Outline • Why Teamware? • What’s Teamware? • Teamware for annotation • Teamware for quality assurance and curation • Teamware for defining workflows, running automatic services, managing annotation projects • Outlook GATE Summer School - July 27-31, 2009

  4. From Annotation Tools to Collaborative Annotation Workflows We have lots and lots of tools and algorithms for annotation; what we need is • methodological instead of purely technological • multi-role instead of single role • assistive instead of autonomous • service-orientated, not monolithic • usable by non-specialists GATE Teamware • Research users in several EU projects • External users at IRF and Matrixware • Interest from other commercial users as well GATE Summer School - July 27-31, 2009

  5. GATE Teamware: Annotation Workflows on the Web GATE Teamware is: • Collaborative, social, Web 2.0, has behaviour mining using Machine Learning • Parallel and distributed (using web services)‏ • Scalable (via service replication)‏ • Workflow based with business process integration GATE Summer School - July 27-31, 2009

  6. Authentication And User Management Workflow Management GATE Annotation Services Teamware – Layer Cake User Interface Layer Language Engineer User Interface Data Curation User Interface Manual Annotation User Interface Document Browser Schema Annotation UI Ontology Annotation UI GATE Developer UI Annotation Diff UI ANNIC UI Teamware Executive Layer Services Layer GATE Document Service GATE Machine Learning API GATE Ontology Service GATE Summer School - July 27-31, 2009

  7. Division of Labour: A Multi-role Methodology • (Human) Annotators - labour has to be cheap! • Bootstrap annotation process with JAPE rules or mixed-initiative learning • Curators (or super-annotators) • Reconcile differences between annotators, using IAA, AnnDiff, curator UI • Manager • Defining annotation guidelines and schemas • Choose relevant automatic services to pre-process • Toolset including performance benchmarking, progress monitoring tools, small linguistic customisations • Define workflow, manage annotators, liaise with language engineers and sys admins • Sys admin • Setup the Teamware system, users, etc. • Language engineer • Uses GATE Developer to create bespoke services and deploy online GATE Summer School - July 27-31, 2009

  8. Teamware: Manual Annotation Tool GATE Summer School - July 27-31, 2009

  9. Manual Annotation Process • Annotator logs into Teamware • Clicks on “Open Annotation Editor” • Requests an annotation task (first button) • Annotates the assigned document • When done, presses the “Finish task” button • If wants to save work and return to this task later – “Save” button, then close the UI. Next time a task is requested, the same document will be assigned, so it can be finished • Depending on the project setup, it might be possible to reject a document and then ask for another one to annotate (Reject button) GATE Summer School - July 27-31, 2009

  10. Hands-on • Open a web browser and Teamware • Login using you user name (from your reg.pack): • <cics-account-id>-annotator • Open the annotation UI • Try requesting tasks, editing annotations, saving your work, asking for another task, etc. • This is what Teamware looks like to a human annotator GATE Summer School - July 27-31, 2009

  11. Teamware for Curators • Still being developed, so UI is in transition • Identify if there are differences between annotators using IAA • Inspect differences in detail using AnnDiff • Edit and reconcile differences if required • New curator UI in Teamware under development • Currently available in Developer GATE Summer School - July 27-31, 2009

  12. IAA: Do my annotators agree? GATE Summer School - July 27-31, 2009

  13. IAA: Results GATE Summer School - July 27-31, 2009

  14. IAA: Recap • The IAA on IE tasks, such as named entity recognition, should be measured using f-measure across all annotators • For classification tasks, use Kappa to measure IAA • For details, see the evaluation lecture and the GATE user guide GATE Summer School - July 27-31, 2009

  15. AnnDiff: Finding the differences GATE Summer School - July 27-31, 2009

  16. Where are these in Teamware? • Only visible to curators and their managers • Resources/Documents menu • Select the corpus worked on • Iterate through each document • Run IAA and AnnDiff, as required • Try for yourself: • Login as <cics-user-name>-curator • Corpus: annie-demo • The first or second documents GATE Summer School - July 27-31, 2009

  17. Forthcoming curator facilities • Have a corpus-level view of IAA • Extended AnnDiff to allow easy reconciliation of the differences between 2 annotators • Currently prototyped in Developer • Will be made available in Teamware soon GATE Summer School - July 27-31, 2009

  18. New AnnDiff in Developer GATE Summer School - July 27-31, 2009

  19. Beyond Pair-wise Reconciliation • AnnDiff only handles 2 sets of annotations at a time – we often need more! • Towards an in-place, content-based reconciliation interface GATE Summer School - July 27-31, 2009

  20. Current UI Prototype GATE Summer School - July 27-31, 2009

  21. Teamware for Managers • Defining workflows • Running annotation projects • Tracking progress GATE Summer School - July 27-31, 2009

  22. Teamware Workflows • Whole process is controlled by a workflow manager • Workflow may be simple: • Give the document to a human annotator • Information curator checks a sample of documents for QC • or more complex • Invoke one or more web services to produce automatic annotations • Pass each document to 2 annotators • Information curator to quickly check level of agreement between the annotators and reconcile any differences • Annotated documents used to train an ML model • When model is good enough, start making suggestions to the annotators GATE Summer School - July 27-31, 2009

  23. Workflow Templates GATE Summer School - July 27-31, 2009

  24. Defining new workflows • Select Projects/WF Templates • Opens the WF wizard • Choose which services you want to run • Choose whether you want manual annotation, how many annotators per doc, … GATE Summer School - July 27-31, 2009

  25. Setting up a Manual Annotation Project • Upload the schemas • Upload the documents • Define the Workflow template • Run the project, choosing the corpus, the annotators, curators, etc. • DEMO! GATE Summer School - July 27-31, 2009

  26. Setting up an Automatic Annotation Project • Configure the web service(s) • Define the Workflow template • Run the project, choosing the corpus • DEMO! GATE Summer School - July 27-31, 2009

  27. Semi-automatic Projects • Just combine the two sets of steps GATE Summer School - July 27-31, 2009

  28. Teamware: Monitoring Project Progress GATE Summer School - July 27-31, 2009

  29. Outlook • Teamware is still under active development • Many features subject to change • If you’d like further information or to try it with your data for a particular project, please contact Hamish and Kalina GATE Summer School - July 27-31, 2009

More Related