Using Mechanical Turk for linguistic research

1. Using Mechanical Turk for linguistic research Fast, cheap, easy, and reliable Tyler Schnoebelen, Stanford University Victor Kuperman, McMasters University

2. The Cloze task • Used by psycholinguists to understandcontextualconstraints • Stimuli 1: Margie ___ • Stimuli 2: Margie moved ___ • Stimuli 3: Margie moved into ___ • Summary: • Turk and students have a highcorrelation (rho=0.823) and performsimilarly (48 stimuli sentences, all with 10+ words)

8. Semantic similarity • Phrasal verbs are sometimes transparent in their meaning: • You lift up a piece of paper and you “lift” and the paper goes “up” • But sometimes opaque: • If you bring up children, the children do not really get brought anywhere (or go up except metaphorically) • We collected judgments on 96 phrasal verbs • Without context, rate them for similarity 1-7 • cool cool down _______________ • Or with context: • The fan will cool down the engine when it starts to heat up. • cool cool down _______________

15. References • Crowdsourcing and language studies: the new generation of linguistic data. Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Los Angeles, CA • Munro, Robert, Steven Bethard, Victor Kuperman, Vicky Tzuyin Lai, Robin Melnick, Christopher Potts, Tyler Schnoebelen and Harry Tily. 2010. • We’ll be presenting at the LSA in Pittsburgh this January, too • Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks, Proceedings of the Conference on Empirical Methods in Natural Language Processing, October 25-27, 2008, Honolulu, Hawaii • RionSnow , Brendan O'Connor , Daniel Jurafsky , Andrew Y. Ng, • "Preventing Satisficing in Online Surveys: A 'Kapcha' to Ensure Higher Quality Data" • Adam Kapelner(University of Pennsylvania) • Dana Chandler