WG2 Task Force “Crowdsourcing”

1. WG2 Task Force“Crowdsourcing” Tobias Hossfeld, Matthias Hirth, Bruno Gardlo, Sebastian Egger, KatrienDe Moor, Patrick Le Callet,Christian Keimel, Klaus Diepold,Valentin Burger WG2 Mechanisms and Models

2. Agenda • Goal of this task force • Problem statement and key applications • Required methodology • Discussion: What are your interests in crowdsourcing? How can we foster collaboration, joint activities? WG2 Task Force „Crowdsourcing“

3. Goals of this Task Force • to identify the scientific challenges and problems for QoE assessment via crowdsourcing but also the strengths and benefits, • to derive a methodology and setup for crowdsourcing in QoE assessment, • to challenge crowdsourcing QoE assessment approach with usual “lab” methodologies, comparison of QoE tests • to develop mechanisms and statistical approaches for identifying reliable ratings from remote crowdsourcing users, and • to define requirements onto crowdsourcing platforms for improved QoE assessment. • Joint activities, collaboration within Qualinet WG2 Task Force „Crowdsourcing“

4. Problem Statement and Key Applications • (Un-)Reliability of users remotely conducting the user tests • Application-layer monitoring, e.g. if browser window has focus • Asking ‘consistency’ questions, questions about content of test • Monitoring of test environment • Technically, e.g. used hardware • Non-technically, e.g. if user gets tired (currently done by analyzing the user results, integrating appropriate questions) • Key applications • Web-based applications, like web browsing, online video streaming, etc. • If specific hardware or software is required, tests may not be possible with crowdsourcing WG2 Task Force „Crowdsourcing“

5. Required Methodology • Test design methodology • Consistency tests, content questions, application-based user monitoring • But not too much to avoid boring the users • Impact of Internet connection must not influence the tests (otherwise we need network monitoring at the end user site) • E.g. downloading the entire test (including videos), • E.g. adaptive streaming, if download is simply not possible (due to the amount of data, live streaming, etc.) • Statistical measures to identify unreliable users • Other tools to check users, e.g. panelcheck http://www.panelcheck.com • Comparison of different QoE testing methods • Lab studies • Crowdsourcing studies / social networking studies (different incentives to participate) • Field trials WG2 Task Force „Crowdsourcing“

6. UniWuE and Crowdsourcing Platforms • Collaboration between Uwue and Microworkers.com provider • We may specify new features  ask for integrating new features into platform • For example, to specify that 50% of the test subjects are younger than 30 years and 50% are older than 30 years • Support / Collaboration with Qualinet partners • introductions for the usage of the Microworkers.com platform, like the account and task creation • help during the task design, which highly affects the result quality. • initial pool of trustworthy workers for QoE tests  pool can be extended and adapted depending on the results of other users QoE tests. • we have ready-to-use hardware to run web based crowdsourcing task and validated mechanisms to integrate these tasks into the microworkers.com platform, like payment-key generation strategies. WG2 Task Force „Crowdsourcing“

7. Advantages of Crowdsourcing / Particular Interests • Building an open QoE panel for Qualinet • More information of users desirable, e.g. profiles from social networks and/or crowdsourcing platforms • Reliable users in panel are preferred for QoE tests, but open to everyone • How to build the panel, how to let it grow? • Allows investigating QoE over time or impact of context on QoE, e.g. same users conduct same QoE tests at several instants in time / different context; see next slide • Combining crowdsourcing and social networks (non-paid CS) • To get users and information about users • To conduct tests with special user groups, “demographic” features • Reliable QoE tests: Test design, Statistical measures, Monitoring • Comparison of different test methods WG2 Task Force „Crowdsourcing“

8. QoE over Time • “QoE over Time” means temporal aspects of QoE / time-dynamic modeling of QoE • Different aspects/viewpoints have to be taken into account • Waiting times: in the beginning (initial delay), during service consumption (stalling) • Single session: short-term memory effects (shown for web-browsing) • Per user: long-term memory effects, expectations are changing over time (2 years ago: 1Mbps; nowadays 16Mbps for DSL users) • Content: long duration videos and QoE testing of this (see NTU) • Beside explicit QoE measures, implicit measures are of interests • E.g. mouse movements (to check reliability, tiredness, etc.) • Cross-activity, e.g. with WG 1 subgroup “Web and cloud apps”

9. Collaboration and Joint Activities • Support by UniWue to setup tests in MicroWorkers.com platform • Update/comment working document sent to WG2 reflector • E.g. features desired for integrating in platforms • STSMs • E.g. joint tests for comparison crowdsourcing and lab • E.g. task design • Joint Activities • Standardization, e.g. updated subjective test methodologies and evaluation wrt. reliability What are your interests?