Best Practices in Longitudinal Usability Studies Mobile Network Operator Perspective Łukasz Nalaskowski

1. Best Practices in Longitudinal Usability StudiesMobile Network Operator PerspectiveŁukasz Nalaskowski

2. Agenda • Longitudinal vs. Lab-based Studies • Longitudinal Studies – MNO's perspective • Case studies • Questions to be answered during the workshop France Telecom Group restricted.

3. Longitudinal vs. Lab-based Studies • Typical evaluation methods (lab-based usability evaluation) tend to focus on the discoverability and learnability problems, rather then usability problems that may persist over time • Long term usability problems are more difficult to evaluate (grasp) but are of the great importance especially in the complex application • It is important to evaluate users' behaviour over time with product and/or services that take longer to learn • Longitudinal studies – any type of studies when data are collected over a few days. In the case of Orange this are mainly dairy studies, where users are asked to report on the usage on the daily basis and the activity is usually concluded with a wrap-up interview France Telecom Group restricted.

4. MNO's Usability Perspective – Why diary studies? • Background: MNOs provide applications and services like mobile internet, game and music applications, mobile TV, voicemail services • The user experience (from MNO point of view) is influenced by a set of 3 factors: • phone and its features: screen size and resolution, keyboard, key pad/ navi-key, soft keys, extra dedicated keys • manufacture implemented OS and phone native UIs: main and sub menus architecture, additional end-user applications (games, music players, internet browsers) • UIs of MNO provided application (games and music downloading application) and content (mobile portal) France Telecom Group restricted.

5. MNO's Usability Perspective: DeviceComplexity • The 3 elements have far reaching implication on the user experience and learning curve • Low end devices (with a small display and weak native internet browser) renders different browsing experience than high-end devices (WAP, web transcoded and full web content) • Low end device does not support certain features, so often mobile portal has to be tailor made to fit particular devices capabilities • Clickable links – some most recent mid range + devices support clickable links in text message bodies, while low end devices don't – visual voicemail case • Bottom line: due to phone complexity it is difficult for a researcher to predict user journey (use cases can be executed differently and different elements of phone hardware and GUIs can interplay with each other) France Telecom Group restricted.

6. MNO's Usability Perspective: Usage Context • Mobile phones are used in various contexts and environments, where all the background circumstances have implications on the usage (Will I use it or will I not use it?) and usage patterns • Background noise • Crowd – limited accessibility, operability, display • Multitasking performed by a user • Rush and time limits • Light and illumination • Bottom line: usage context and environment is difficult to predict and replicate in the lab France Telecom Group restricted.

7. MNO's Usability Perspective – Why diary studies? • The complexity of the mobile phones and variety of usage contexts cast doubts on the reliability of lab based usability testing as a main method of user experience evaluation. • Hence the traditional lab based usability studies are usually accompanied by additional methods of data gathering which reach deeper into particular user experience. • The two most commonly used are diary studies and different forms of ethnography. The former can be recognized as one of a type of longitudinal study. France Telecom Group restricted.

8. Case studies: Game downloading application • Background: Orange planned to launch a game downloading application. The research goal was to understand mobile phone gamers' usage habits (needs gathering study) • Methodology: A quick 5 day diary study was executed. The diary study was concluded with lab-based usability sessions • Findings: • Mobile users tend to refer to mobile portal for game downloading than dedicated application embedded in the phone menu • Playing a game on a mobile – a time filler, when bored, on public transport • A moment to while away/ spend some time on the bus • Implications: different mindset (mobile portal as a place to look for games); the application has to be quick and easy to initiate – with the least possible number of clicks; downloading time has to be short France Telecom Group restricted.

9. Case Study: Business Voicemail • Background – Orange is building a voicemail solution for business users. In order to better understand the current usage of voicemail a need gathering study was launched • Although voicemail frequently doubts whether during the lab based session the user experience feedback would be elicit • This might prove difficult for users to memorize all the interesting cases when they used voicemail and the problems and difficulties that they encountered when using it • Hence a decision to conduct a diary study • Methodology: A selected group of business voicemail users were asked to keep a track of their voicemail usage on the daily base on a span of 5 working days • Key findings: a direct access to particular voicemail message is very much need. With the traditional voicemail accessed via IVR, users have to listen to the messages in chronological order, as they were deposited. In their busy environment they would like to see who has called them, when and select the messages that they want to listen to in first. France Telecom Group restricted.

10. Questions to be answered • What are the reasons for conducting longitudinal usability studies instead of standard lab-based studies? • The diversity of usage context • Unpredictability of usage context • In what cases lab-based, task-based standard methodologies seems to fail to provide a reliable feedback? • Complex services where different UIs are interrelated and intertwined within one system (mobile phone) • What services (mash-ups case)? • When users swap between services providers and had to learn a new service? • Previous experience (mindset) can work both as an enhancer as well as a barrier/ obstruct the usage France Telecom Group restricted.