Human Computation Play a Game to Develop an Ontology

1. Human ComputationPlay a Game to Develop an Ontology Peyman Nasirifard p+e+y+m+a+b-b+n dot sin(arcsin(lastname)) @ deri.org

2. Agenda • Introduction • CAPTCHA • Games with a purpose • ESP game • Peekaboom • Verbosity • Possible game for developing simple ontologies • Play a game • Conclusion

3. Introduction • Human-based computation is a technique when a computational process performs its function via outsourcing certain steps to humans.

4. Back to History • Yahoo! and Gmail are not interested to enable a bot to create thousands accounts per day for sending spam • They use CAPTCHA to prevent it plus

5. CAPTCHA • Stands for “Completely Automated Public Turing test to tell Computers and Humans Apart” • Luis von Ahn et al. coined the term in 2000 • A Program that can tell whether a user is a human or a computer • Many different techniques

6. Some Examples

7. Dog or Cat? Human: mmm… dog Computer: mmmmmmmmmmmmmmm…

8. Human Computation • If we use people to break CAPTCHA, we are doing human computation • In some countries, some companies hire people to break CAPTCHA and send spam • Some companies cleverly use humans to break CAPTCHA and send spam • How?

9. Clever spammers

10. Clever Spammers Free Nude Photos Type the word in the box if you want to see the next picture

11. Really?! • Jan 2004: world without spam by 2006! • Huge amount of investment • Bill Gates receives 4 million spams per day

12. Nice Quote • Luis von Ahn: Instead of hiring people and pay them to solve our problems, we can design games and people will pay us to play our games and solve our large-scale problems!

13. The ESP game • Object of the game: type the same word • Only thing in common is: an image • Players • Do not know each other (randomly paired) • Can not communicate • Advantages: • Two different sources labels the image • enjoyable • labels all images on Google image in a short time • Help to improve English! • There are many people that play over 20 hours a week

14. The ESP game Player 1 Player 2 • WOMAN • CAR • CAR • GIRL • TREE Agree: CAR Get points

15. Taboo words • Taboo words • More difficult, but more fun • CAR • WOMAN

16. Single version of ESP game • WOMAN • CAR • CAR • GIRL • TREE • The engine records everything from previous players • A single player will actually play with another player, but not at the same time

17. Cheating and Repetition • Problem: Agreement on cheating • Let’s label all images with “dog” • Solution: At random, system gets players test images to check whether they play honestly or not • If they do not play honestly, the system will let them play, but nothing will be recorded • For certainty, only labels which at least N pairs agreed upon will be stored

18. The Limitations of ESP • The ESP Game can label images (and consequently tell you what’s in them), but it cannot: • Find the objects being labelled • Determine the way in which the object appears – does the label “car”refer to the text “car” or an actual car in the image?

19. The place of objects in an image • Such information would be extremely useful for computer vision research man dog

20. The Guesser guesses: • Flower • Petal • Butterfly The Revealer clicks on parts of the image and shows them to the Guesser. Server: Correct, Butterfly

21. Hints The label “car” is ambiguous -- this is “car” this is also “car” The hints help distinguish the manner in which the label “car” appears: this is the object “car” this is the text “car”

22. Verbosity • Collect common-sense facts • Water quenches thirst • Sky is blue • Lions eat meat • We as human know hundreds of millions common sense facts • Computers do not know • If know, potentially make them more intelligent (e.g. search better)

23. Common sense fact samples • Milk • It is liquid • It is white • it has lactose • cereal is eaten with it

24. Verbosity Narrator MILKis typically near cereal is a liquid Guesser

25. Verbosity Guesser is typically near cereal is a liquid MILK Narrator MILK

26. Verbosity Narrator Object Common sense facts about the object Guesser

27. Verbosity Narrator Object Guesser Common sense facts about the object Object

28. Templates • • ___ is a kind of ___. Allows for hierarchical categorization. • • ___ is used for ___. Provides information about the purpose of a word. • • ___ is typically near/in/on ___ (three templates). Provide spatial data. • • ___ is the opposite of ___ / ___ is related to ___ (two templates). Provide data about basic relations between words. • • ___. In the game, this is a “wildcard” that collects related words.

29. Symmetric vs. Asymmetric • Verbosity is a asymmetric game, whereas ESP game is a symmetric game. • Symmetric games: constraint is number of outputs per input • Asymmetric games: constraint is number of inputs that produces the same output

30. Possible game to build an ontology • Several game should work together • Images come from ESP game • Not always: only those images are selected which have one object in it • i.e. car, bike, monitor, mouse, house • These images are input to next game which tries to catch the properties of objects • car has colour, car has wheels, car has manufacture, car has owner, car has building year, etc.

31. Possible game to build an ontology • Cardinality will be caught by templates, as soon as we have properties. • Car has four wheels • Car has one plaque • These sentences will be transferred to OWL representation using a mediator. • The more pairs play the game, the more complex the ontology will be

32. Contact me if you are interested to work on it

33. This page intentionally left blank

34. Guess what! • It has usually four wheels • It has usually one seat • It is kind of vehicle • It operates with human power • It operates with batteries • It has a break system • It is a kind of chair

35. Answer

36. Conclusion • Games are enjoyable! • More than 9 billion Human-hours of solitaire are played each year • We may cleverly using humans to solve large-scale problems by designing interesting games • Many people play word-guessing games to improve their English • Go and play to promote science!

37. References [1] Verbosity: A Game for Collecting Common-Sense Facts, http://www.cs.cmu.edu/~biglou/Verbosity.pdf [2] Peekaboom: A Game for Locating Objects in Images, http://www.cs.cmu.edu/~biglou/Peekaboom.pdf [3] Labeling Images with a Computer Game, http://www.cs.cmu.edu/~biglou/ESP.pdf [4] Games with a Purpose, http://www.cs.cmu.edu/~biglou/ieee-gwap.pdf [5] Wikipedia, http://en.wikipedia.org/wiki/Human-based_computation [6] We'll End Spam Within 2 Years, http://www.connectedhomemag.com/Networking/Articles/Index.cfm?ArticleID=41587http://news.bbc.co.uk/2/hi/business/3426367.stm [7] CAPTCHA, http://en.wikipedia.org/wiki/Captcha, http://www.captcha.net [8] ESP game, www.espgame.org [9] Peekaboom game, http://www.peekaboom.org/ [10] Verbosity game, www.peekaboom.org/verbosity/ [11] Presentation, http://isandtcolloq.gsfc.nasa.gov/fall2006/presentations/Ahn.ppt [12] Presentation, www.aladdin.cs.cmu.edu/workshops/lamps05/Slides/Peekaboom.ppt

38. Game Over p+e+y+m+a+b-b+n dot sin(arcsin(lastname)) @ deri.org