1 / 32

Harnessing manpower for creating semantics (doctoral dissertation)

Institute of Informatics and S oftware Engineering , Faculty of I nformatics and I nformation T echnologies , Slovak University of Technology in Bratislava. Harnessing manpower for creating semantics (doctoral dissertation). Jakub Šimko jsimko @ fiit.stuba.sk.

evette
Télécharger la présentation

Harnessing manpower for creating semantics (doctoral dissertation)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Institute of Informatics and Software Engineering, • FacultyofInformaticsand InformationTechnologies, • Slovak UniversityofTechnology in Bratislava Harnessing manpower for creating semantics(doctoral dissertation) JakubŠimko jsimko@fiit.stuba.sk Supervised by: prof. Mária Bieliková July 4th, 2013

  2. Games with a purpose (GWAP) for semantics acquisition

  3. Games with a purpose • Cheap (once they are created) • Difficult to create [Quinn & Bederson. Human computation: a survey and taxonomy of a growing field. CHI’11, 2011]

  4. ESP Game: image metadataacquisition What is in the image? Player 1: Player 2: water sky bridge Mostar night river bridge Bosnia The players must blindly match Banned words: blue, towers [Von Ahn & Dabbish: Designing games with a purpose. Commun. ACM, 2008.]

  5. Motivation • Open issues in semantics acquisition • Modelling of specific domains • Personal multimedia metadata acquisition • Metadata upkeep • Games with a purpose (GWAPs): design issues • In general: no design methodology (young problem area) • Cold start problems • Quality management, effectiveness of work allocation

  6. Thesis Goals • Create new, GWAP-based approaches to semantics creation, particularly for specific domains • Bring in generally applicable improvements to GWAP design, focusing on selected problems

  7. Work overview State of the art: • GWAP taxonomy and design space GWAPs we created: • LittleSearch Game: term network acquisition • PexAce: (personal) imagery tag acquisition • CityLights: validation of music metadata General GWAP design improvements: • Helperartifacts: coldstartproblemreduction • Playercompetences: improving GWAP output quality

  8. Our taxonomy of GWAPs

  9. GWAP design • A relatively new area (<10 years) • No holistic design methodology exists • GWAPs are created ad-hoc • Few works aimed at particular design issues • [Ahn, 2008] Player agreement schemes • [Chiou, 2011] Suggested considering player skills in GWAPs • Our contribution: GWAP design dimensions • following the idea of design lenses [Schell, 2008] [Von Ahn & Dabbish: Designing games with a purpose. Commun. ACM, 2008.] [Chiou & Hsu. Capability-aligned matching: improving quality of games with a purpose. AAMAS ’11] [J. Schell. The art of game design a book of lenses. Elsevier/Morgan Kaufmann, 2008.]

  10. Our GWAP design dimensions

  11. Existing GWAPs in our design space

  12. PexAceGoal: acquire (personal) image tagsNew artifact validation modelQuality management through player modelling • International Journal on Human-Computer Studies [In press] • Šimko, J., Tvarožek, M., Bieliková, M. Human Computation: Single-player Annotation Game for Image Metadata. • SMAP 2011 (IEEE CS Press) • Šimko, J., Bieliková, M.: Games with a Purpose: User Generated Valid Metadata for Personal Archives. • I-Semantics 2012 (ACM) • - Šimko, Jakub - Bieliková, Mária: Personal Image Tagging: a Game-based Approach. I-Semantics, 2012

  13. PexAce: acquisition of image metadata • Cards– image pair seeking memorygame • Players create image annotationsto aid their memory

  14. PexAce: general domain deployment • (Standard) Corel 5K dataset: photos + tags + our tags • 107 players, 814 games, 2 792 images • 22 176 annotations, 5 723 tags • Golden standard comparison: 73% precision • Aposteriori evaluation: 94% precision • Automated methods ~70% * • Limited set of tags *[Duygulu et. al. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary 2002. Springer-Verlag.]

  15. PexAce for personal images • Personal image metadata – virtually impossible to get • Personal images instead of general images in PexAce • Players like that more • They provide specific annotations (metadata) • Experiments: 2 x 2-player groups, 50 images each • Correctness: 94% • 44% specific tags Other (11%) Places (15%) Events (21%) Persons (53%)

  16. „Benevolent“ artifactvalidation model Annotations decomposed to votes: P - players, T- terms, I - Images Original mutual player supervision Less strict heuristics

  17. Artifact validation and cold start problem:A general GWAP issue „Howcan a resultofahumanintelligencetaskbeautomaticallyevaluated?“ GWAPs use: • Approximative or exactautomatedevaluation (case dependent) • Mutualplayersupervision Threat to multiplayer validation schemes: COLD START ‘’The requirement is to have multiple players online at the same time, sometimes with a requirement that they cannot communicate.” Keep the games single-player

  18. Helper artifacts: a new artifact validation principle Helperartifacts: • Decouplescoringfromtasksolving, insteadmotivateplayers to solve tasks to help themselves in the progress of the game • E.g. in PexAce, a player may win the game well enough even without the annotations • Potential of general applicability (to any existing game)

  19. Quality management in GWAPs:Considering differences inplayer competences • Quantify player skills – player model (e.g. player’s task-solving expertise for each sub-domain) • Apply model in • “post-processing” - Solution filtering (e.g. vote weighting) • “pre-processing” - Task assignment (e.g. match task subdomain to expertise areas) • Speed up the process or/and retrieve higher quality results

  20. Measuring player competences: PexAce data • Usefulness (delivery of correct artifacts) • Consensusratio (agreement with other players) • Correlation: 0.496

  21. Little Search GameGoal: acquire lightweight term networkstatistically unsupported, yet valid term relationshipsspecific domain use Int. J. on SemanticWeb and InformationSystems -Šimko, Jakub - Tvarožek, Michal - Bieliková, Mária: SemanticsDiscoveryviaHumanComputationGames. In: InternationalJournal on Semantic Web and InformationSystems (2011) Hypertext 2011 (ACM) -Šimko, Jakub - Tvarožek, Michal - Bieliková, Mária: LittleSearch Game: Term NetworkAcquisitionvia a HumanComputation Game. Hypertext, 2011

  22. Little Search Game (negativesearch game) • Creation of lightweight term network • Player’s task: reduce number of results with negative search Searchquery: „Star –movie –war –death“

  23. LSG Term network evaluation • Aposteriori evaluation: 91% correctness • A potential to add term relationships to existing bases • 59% of LSG rels. do not exist in ConceptNet* corpus • …including demanded non-taxonomic relationships *[Liu & Singh. ConceptNet— A Practical Commonsense Reasoning Tool-Kit. BT Technology Journal 2004]

  24. Hidden term relationships – hard for automated discovery (40% of LSG term network)

  25. LSG modification: TermBlaster(Harvesting relationships for software design domain) • Specific domain • No text typing • 71 % correct, 21% „hidden relationships“

  26. CityLightsGoal: validate existing music tagsquality management through confidence expression I-Semantics 2012 (ACM) -Dulačka, Peter - Šimko, Jakub - Bieliková, Mária: ValidationofMusicMetadatavia Game with a Purpose. I-Semantics 2012

  27. CityLights: music tag validation(a concept of validation question) Tag support value: + increases + player selects the group • decreases • - p. doesn’t select the group • - player rules out the tag Validation question: “Which of these tag groups characterizes the music track you hear?” • Rockabilly, USA, 60ties • Seasonal, rich oldies, xmas • February 08 love, oldies, 60 musik • Wrong and correct tags bubble out • Possitive and negative thresholds

  28. CityLights: experiments • LastFMdataset • 875 games, 4933 questions, 1492 tags • Feedback actions per tag: • 17.75 implicit • 5.29 explicit • Optimized parameter configuration • 68% correctness

  29. Betting mechanism: Measuring competence through confidence • Betting mechanism within a GWAP • Through bet height, the player expresses his confidence in his task solution • CityLights case: bet height aligns with impact on tag validity value • Helps with cold start problem associated with user modeling

  30. Main contributions • Definition GWAP design space • GWAPs for semantics acquisition • For specific domains (personal images, SW engineering) • For otherwise hardly discoverable semantics (hidden rels.) • New GWAP design principles • Helper artifacts for cold start reduction • Metrics for long term player competence modeling • Betting mechanism for short term player competence acq. • Metadata validation GWAP concept

  31. Summary • GWAP taxonomy and design dimensions • [survey paper prepared] • Little Search Game – Lightweight term network acquisition • Hidden term relationships • Hypertext 2011, ACM • Int. J. of Semantic Web and Information Systems, 2011 (CC, IGI) • PexAce – Personal image metadata acquisition • Helper artifacts • Competence measures • SMAP 2011, IEEE • I-Semantics 2012, ACM • Int. J. of Human-Computer Studies, 2013 (CC, Elsevier) • CityLights – Music metadata validation • Betting mechanics – player competence through confidence • I-Semantics 2012b, ACM

  32. Selected publications • SemanticsDiscoveryviaHumanComputationGames. In: InternationalJournal on Semantic Web and InformationSystems. 2011 • HumanComputation: Single-playerAnnotation Game forImageMetadata. InternationalJournal on Human-ComputerStudies. 2012 [In press]. • ValidationofMusicMetadatavia Game with a Purpose. I-Semantics 2012 (ACM) • Gameswith a Purpose: UserGeneratedValidMetadataforPersonalArchives. SMAP 2011 (IEEE CS) • LittleSearch Game: Term NetworkAcquisitionvia a HumanComputation Game. Hypertext 2011 (ACM) • PersonalImageTagging: a Game-basedApproach. I-Semantics 2012 (ACM)

More Related