Human Computation & ESP Game

1. Presenter: Lin, Sin-Yan Human Computation & ESP Game 2008/12/19

2. References • Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems. • Ahn, L. V., Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems. • Ahn, L. V. (2007) Human computation, paper presented at the Proceedings of the 4th international conference on Knowledge capture. • Ahn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8), 58-67. • Chen, L.-J., Wang, B.-C., Chen, C.-Y., King, I. & Lee, J. (2008) An analytical study of puzzle selection strategies for the esp game. Report for Institute of Information Science, Academia Sinica (Taipei, Taiwan, ROC). • Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the esp game. Report for Microsoft Research, Microsoft Corporation.

3. Outline • Human Computation • Human Computation System and Application • Human Computation in E-Learning • The ESP Game • Mathematical Model in ESP Game • System Gain • Image selection • Automatic playing the ESP Game • Problems of ESP Game • Conclusion and Comments

4. Human Computation • Concepts are started at 2001: The Open Mind Initiative • Some task are trivialfor humans, but continue to challenge computer algorithms • Treat human brains as processors in a distributed system, each can perform a small part of a massive computation • But humans require some incentive to become part of a collective computation • By Games (Ahn and Dabbish, 2004) (Ahn and Dabbish, 2008) Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems Ahn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8), 58-67.

5. Human Computation System- Peekaboom Ahn, L. V., Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems.

6. Human Computation System- PhotoShoot (1/2) Cheng, K.-Y. (2006). PhotoShoot: A Web-Game for User Assisted ROI Labeling. Information Management. Taipei, Taiwan, National Taiwan University. Master.

7. Human Computation System- PhotoShoot (2/2)

8. Human Computation System- ImageHunter Chen, L.-H. (2008). Accuracy Analysis in Productivity Games: A Case Study on Landmark Annotation. Department of Computer Science and information Engineering. Taipei, Taiwan, National Taiwan University. Master: 62.

9. Human Computation System- PhotoSlap Chang, T.-H. (2007). Productivity Game Design and Gameplay Analysis: Play and Annotate with PhotoSlap. Department of Computer Science and Information Engineering. Taipei, Taiwan, National Taiwan University. Master: 52.

10. Human Computation System- reCAPTCHA • Proposed by Ahn (2008) • Gathering the characters that optical character recognition (OCR) software has been unable to read

11. Human Computation in E-Learning • 人力計算具有利用人類直覺的特性 • 動機 • 傳統英文閱讀學習及評量無法達成的地方 • 目的 • 用TAGs表達英文閱讀學習過程中最直覺的想法 • 方法 • Human Computation and Game Design

12. Human Computation in E-Learning- why tag • Free, uncontrolled vocabulary • Social relations of TAGs image photo color design tagging art research

13. Agenda • Human Computation • Human Computation System and Application • Human Computation in E-Learning • The ESP Game • Mathematical Model in ESP Game • System Gain • Image selection • Automatic playing the ESP Game • Problems of ESP Game • Conclusion and Comments

14. The ESP Game • The first human computation system to take advantage of people’s desire to be entertained and provide useful metadata Randomly selected image Input Keywords Input Keywords Keyword Match Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems

15. Mathematic Model in ESP Game • There are many parameters in ESP Game • We can use those parameters to modeling many things that we want to know • Ex. • Number of players • Number of labels per game • Number of guesses to reach a consensus • Good and badword assumption • (Human Computation) System Gain • Image selection • User scoring model

16. Mathematic Model in ESP Game- System Gain Modeling (1/2) • The performance of human computation system • The purpose of ESP Game tries to collect images’ tags • Two aspect of thinks • Take as many tags as possible for each image • Playing the maximum number of image • A metric to evaluate the system gain

17. Mathematic Model in ESP Game- System Gain Modeling (2/2)

18. Mathematic Model in ESP Game- Image Selection (1/3) • Different image selection method • Random • Fresh-first • Maximize ln(N) • Optimal (OPSA) • Largest possible system gain • P0: all images that have not been played • P1: all images that have been played at least once, but less than r times • P2: other images

19. Mathematic Model in ESP Game- Image Selection (2/3) T / N

20. Mathematic Model in ESP Game- Image Selection (3/3) M: Total number of images in the system Wang, B.-C., C.-W. Lin, et al. (2008). An Analytical Study and Modeling of the ESP game. Taipei, Taiwan, Institute of Information Science, Academia Sinica.

21. Mathematic Model in ESP Game- Automatic Playing the ESP Game (1/4) • Given enough instances of labeled images as training data • Similar with Language model • Assign probability to the next label to be added • Plays the ESP Game without looking at the image • Limitation: at least one “off-limits” in the image Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

22. Mathematic Model in ESP Game- Language Modeling (1/2) • Model the sequence of word usage of a certain language • Given a Language Model: M • P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this)

23. Mathematic Model in ESP Game- Language Modeling (2/2) • P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this) • Markov assumption: • To give reasonable predictions, only the prior local context (the last few words) affects the next word • n-gram word model • Bi-gram: • P(I love this game) = P(I) P(love|I) P(this|love) P(game|this)

24. Mathematic Model in ESP Game- Automatic Playing the ESP Game (2/4) • Given enough instances of labeled images as training data • At least one “off-limits” in the image • t: next label • T: off-limits set • P(“t is next label” | “set T already present”) Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

25. Mathematic Model in ESP Game- Automatic Playing the ESP Game (3/4) • P(“t is next label” | “set T already present”) • Conditional independence assumption • The problem of zero probability Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

26. Mathematic Model in ESP Game- Automatic Playing the ESP Game (4/4) • The problem of zero probability Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

27. Mathematic Model in ESP Game- Others • User scoring model • More informative tags

28. Problems of ESP Game • There is a lot of redundancy in the tag sets • Even when tags are not exactly synonyms, they are often “to be expected” given the other tags • There is a tendency to match on colors • People tend to add more generic labels Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

29. Conclusion and Comments • The first thing of Human Computing research is finding a clear PURPOSE • How your game or method to achieve the purpose