1 / 15

Using Mechanical Turk for linguistic research

This study explores the use of Amazon's Mechanical Turk as a tool for linguistic research, focusing on cloze tasks and phrasal verb semantics. Researchers Tyler Schnoebelen and Victor Kuperman highlight the efficiency of crowdsourcing to gather linguistic data, showing a strong correlation (rho=0.823) between Turk participants and student performance on various stimuli. The reliability and effectiveness of non-expert annotations are evaluated, making it clear that Mechanical Turk offers a promising avenue for collecting high-quality linguistic insights quickly and affordably.

lilike
Télécharger la présentation

Using Mechanical Turk for linguistic research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using Mechanical Turk for linguistic research Fast, cheap, easy, and reliable Tyler Schnoebelen, Stanford University Victor Kuperman, McMasters University

  2. The Cloze task • Used by psycholinguists to understandcontextualconstraints • Stimuli 1: Margie ___ • Stimuli 2: Margie moved ___ • Stimuli 3: Margie moved into ___ • Summary: • Turk and students have a highcorrelation (rho=0.823) and performsimilarly (48 stimuli sentences, all with 10+ words)

  3. Semantic similarity • Phrasal verbs are sometimes transparent in their meaning: • You lift up a piece of paper and you “lift” and the paper goes “up” • But sometimes opaque: • If you bring up children, the children do not really get brought anywhere (or go up except metaphorically) • We collected judgments on 96 phrasal verbs • Without context, rate them for similarity 1-7 • cool cool down _______________ • Or with context: • The fan will cool down the engine when it starts to heat up. • cool cool down _______________

  4. References • Crowdsourcing and language studies: the new generation of linguistic data. Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Los Angeles, CA • Munro, Robert, Steven Bethard, Victor Kuperman, Vicky Tzuyin Lai, Robin Melnick, Christopher Potts, Tyler Schnoebelen and Harry Tily. 2010. • We’ll be presenting at the LSA in Pittsburgh this January, too • Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks, Proceedings of the Conference on Empirical Methods in Natural Language Processing, October 25-27, 2008, Honolulu, Hawaii • RionSnow , Brendan O'Connor , Daniel Jurafsky , Andrew Y. Ng, • "Preventing Satisficing in Online Surveys: A 'Kapcha' to Ensure Higher Quality Data" • Adam Kapelner(University of Pennsylvania) • Dana Chandler

More Related