Outline

1. Quality of Service and Quality of Experience Correlations in a Location-Based Mobile Multiplayer Role-Playing Game Maarten Wijnants1, Wouter Vanmontfort1, Jeroen Dierckx1, Peter Quax1, Wim Lamotte1, Katrien De Moor2, Jeroen Vanattenhoven31Hasselt University – Expertise Centre for Digital Media (EDM) - IBBT {firstname.lastname}@uhasselt.be2Ghent University – Media and ICT (MICT) - IBBTKatrienR.DeMoor@Ugent.be 3KULeuven - Centre for User Experience Research (CUO) - IBBT jeroen.vanattenhoven@soc.kuleuven.be

2. Outline • Introduction and Motivation • Research Context • Location-Based Role-Playing Game • Investigated Technological Factors • User Research • Practical Approach and Participant Demography • Test Procedure and Data Collection Methodology • Quantitative and Qualitative Findings • Conclusions ICEC2011

3. Introduction and Motivation • Quality of Service (QoS) • Well-established concept in software and system engineering • Measure of technological performance • E.g., performance guarantees in computer network • Technical dimensions  Objectively measurable • Quality of Experience (QoE) • Fairly novel active research topic • Semantic variant of QoS • Denotes overall experience or satisfaction of end-user • Humane, multi-disciplinary and highly subjective metric and therefore hard to quantify objectively ICEC2011

4. Introduction and Motivation • QoS and QoE are irrefutably related • Strong attention for technological factors remains explicit in user research • However, exact connection to date remains largely uninvestigated and therefore unclear • Such information is highly valuable for developers of (distributed) applications • Decent understanding of their mutual impact and interaction likely enables user experience optimization • Important asset in Customer Relationship Management • This paper unveils QoS/QoE correlations via a hybrid user research methodology ICEC2011

5. Research Context • The field of mobile gaming was selected as context to conduct our research • Increasingly popular application domain • Focus on distributed multiplayer games • Network operation and performance might substantially influence user experience • Further scope reduction by concentrating on (outdoor) location-based games • Revolve around location determination: Position of players/objects in the physical world is tracked and plays prominent role in gameplay ICEC2011

6. Research ContextLocation-Based Role-Playing Game • In-house developed mobile location-based multiplayer game with WW I setting • Played outdoors between two rivaling teams (Allied forces and German army) • Location-based game • Players walk around in the physical world while carrying a handheld device (netbook) • Player location tracking via GPS receiver • Data dissemination via 3G connection • Game presents satellite view of the outdoor environment ICEC2011

7. Research ContextLocation-Based Role-Playing Game • Interaction occurs by “scanning” QR tags • Each player carries (unique) QR code • Codes in playing area represent virtual items • Role-Playing Game (RPG) • Profile determines actions player can undertake, his objectives and type of collectable virtual items • Commander: Remains at a fixed indoor location; has a global overview of his team’s actions, permanently sees discovered objects and can instruct players on the field • Soldier: Collect weapons; engage enemies • Spy: Collect intelligence items; reveal enemy players • Medic: Collect medical supplies; heal injured teammates ICEC2011

8. Research ContextLocation-Based Role-Playing Game ICEC2011

9. Investigated Technological Factors • 3 technical/QoS-related parameters studied for their impact on player QoE • Each parameter acted as separate test condition • Parameter 1: Player Location Determination Precision • Investigate impact of imprecisions on player QoE • Position of players artificially and randomly varied in 10 meter radius around actual location • Parameter 2: Player Location Refresh Rate • Important design variable for location-based apps • Refresh player locations only per 6 seconds ICEC2011

10. Investigated Technological Factors • Parameter 3: Voice communication delay • Communication is affected by network delay • ITU recommends 1-way latency < 400 ms • 2 seconds of additional artificial delay • Test conditions are representative, but the used magnitudes might be exaggerated • To elicit pronounced and unambiguous reactions from test subjects ICEC2011

11. User ResearchPractical Approach and Participant Demography • 4 separate user studies with 8 subjects each • Participants evenly divided in 2 teams • Each participant assumed different role (random) • Deliberately not informed about QoS manipulation • Test population demography • 12.5 % was female; average age 29 (SD = 9.04) • Over 80 % had played video game in past month • Professional occupation: mostly students (40 %) and employees (50 %) • 7000 m2 science park as playing area • 25 virtual items (QR codes) strategically dispersed ICEC2011

12. User ResearchTest Procedure and Data Collection Methodology • Test session of approximately 2 hours, logically enveloping 3 consecutive phases • Pre-usage phase • Game objectives and mechanics verbally outlined • Paper-based questionnaire • Basic socio-demographical characteristics • Previous digital gaming experiences and expectations • During-usage phase • 40 minute duration; 4 intervals of equal length • Three intervals exclusively devoted to 1 test condition • Investigate influence of corresponding QoS parameter • 1 interval with “perfect” gameplay (reference scenario) ICEC2011

13. User ResearchTest Procedure and Data Collection Methodology • In course of play session, 4 identical in-game surveys needed to be electronically filled in • At end of each conceptual interval of the session • Acquire detailed QoE data in each test condition • In-situ data sampling at the very moment of experience • Reduces risk of recall bias effect  Accurate assessments • 11 statements measured on 5-point Likert scale • Post-gaming phase • Second paper-based questionnaire • Inquiries about overall experiences, feelings, thoughts • Participant group conversation (focus group) • Share experiences, reflect on (non-)reported QoE factors ICEC2011

14. User ResearchQuantitative and Qualitative Findings Mean scores for in-game survey statements per test condition Global characteristics of non-instrumental dimensions • Users felt least ‘absorbed’ in role in interval 1; feeling monotonously increased as game progressed (same for ‘fun’) • Users needed to get used to the game? • Expectations adjusted in course of the gaming session • Pre-usage expectations were high; adapted according to context ICEC2011

15. User ResearchQuantitative and Qualitative Findings Mean scores for in-game survey statements per test condition Player location distortion test condition • Accuracy of location-related information concerning other players as well as objects was rated lowest • Influenced users’ perception of location update rate (both players and objects)  Unjust! ICEC2011

16. User ResearchQuantitative and Qualitative Findings Mean scores for in-game survey statements per test condition Voice delay test condition • Participants felt most ‘frustrated’ + most ‘effort’ required • High rating for ‘experiencing technical problems in the communication with team members’ + lowest average score for ‘I can communicate smoothly with my team members’ ICEC2011

17. User ResearchQuantitative and Qualitative Findings Mean scores for in-game survey statements per test condition Normal gameplay test condition (no QoS tampering) • Highest degree of ‘absorption’ in the played role + maximal score for ‘fun’ • Highest rating for accuracy of player location information + for position refresh rate of both players and objects ICEC2011

18. User ResearchQuantitative and Qualitative Findings Mean scores for in-game survey statements per test condition Limited player location refresh test condition • Minimal ‘frustration’ and required ‘effort’ + high score for ‘fun’ • Technical manipulation no notable influence on QoE • Claim is supported by qualitative data from the post-gaming stage • Players were near the end rather centrally clustered (entire playing area already discovered)  less reliance on the data displayed on their device? ICEC2011

19. User ResearchQuantitative and Qualitative Findings • During-usage study succeeded in relating modifications in technical circumstances to the users’ (subjective) experiences • Example results from post-usage stage • QoS manipulation in 1st and 2nd playing condition were experienced as sources of frustration • Consistent with findings from in-game surveys • Quotes from focus groups • “The GPS update rate wasn't really a problem; most of the time they weren't running, so you could easily track the other players” • “As Commander I expect to know the precise location of my team; the faulty positions made this impossible” ICEC2011

20. User ResearchQuantitative and Qualitative Findings Mean scores for in-game survey statements per player role Did player role influence QoE? • Statistically significant differences between roles for 5 aspects • ‘Absorption’: Soldiers felt notably more immersed than Commanders • ‘Fun’: Soldiers had more pleasure than the Medic and Commander roles • ‘Effort’: Significantly higher for Soldier profile (esp versus Commander) • ‘Expectations’: Opposite of ‘effort’ (Commander > Soldier) • Player location update rate: Clearly lowest for Commander (cfr post-usage) ICEC2011

21. Conclusions • Two high-level conclusions • Modifications in 3 specific technical parameters held implications for quality of users’ experiences • E.g., test condition without QoS impairments had highest average scores for ‘absorption’ and ‘fun’ • For some sub-dimensions, player reported QoE diverged depending on assumed role • E.g., maximal ‘absorption’ and ‘fun’ for Soldier profile • Complementary qualitative and quantitative data ICEC2011

22. Conclusions • Lower-level, specific findings illustrate extent of impact of QoS performance on QoE • E.g., voice communication latency complicated coordination of players’ actions • Low ‘absorption’ + highest mean ‘frustration’ and ‘effort’ • Valuable information for software developers • How best to cope with QoS irregularities? • ‘Absorption’, ‘fun’ monotonously improved while playing + users adjusted expectations • QoE not merely determined by technical params • QoE exceeds QoS in terms of scope; importance and potential influence of non-instrumental aspects on QoE ICEC2011

23. Questions ICEC2011

24. User ResearchQuantitative and Qualitative Findings • Findings from pre-usage surveys • Covered (among other things) same 11 QoE-related statements as the in-game surveys • Pre-usage responses indicated that test subjects attached most importance to • “Having fun” (Mean = 4.58) • “Not having technical problems while communicating with team members” (M = 4.42) • “Being able to communicate smoothly with team members” (M = 4.23) • Analysis of post-usage data resulted in the same top 3 list ICEC2011

25. In-Game Questionnaire I feel totally absorbed in my role in the game With the information that I am receiving about the position of the other players, I am able to localize them without a problem I am having fun while playing the game I have to put a lot of effort in the game The kind of information that I am getting about the location of objects, allows me to localize these objects without a problem I expected more from the game While communicating with team members I experienced technical problems I feel frustrated The location of the other players is updated sufficiently I am able to communicate smoothly with my team members The location of objects is updated sufficiently ICEC2011