Download
1 / 32
320 likes | 324 Vues
IN4MATX 231 Human-Computer Interaction. Observing Users. Presenters: Ritesh Subramanian Tanmay Goel. Outline. Benefits and challenges of different types of observations. How to observe as an onlooker, a participant, and an ethnographer.
E N D
IN4MATX 231 Human-Computer Interaction Observing Users Presenters: Ritesh Subramanian Tanmay Goel
Outline • Benefits and challenges of different types of observations. • How to observe as an onlooker, a participant, and an ethnographer. • How to collect, analyze and present observational data. • Indirect observation using diary studies and logging. • Qualitative and Quantitative analysis.
What and when to observe • Observation is usable at all stages during product development. • Goals and questions determine the paradigms and techniques used. • Observers can be: • Onlookers • Participants • Ethnographers • The degree of immersion that evaluators adopt varies across a broad outsider-insider spectrum.
Approaches to Observation • “Quick and Dirty” observation • It can occur anywhere, anytime • Least formality involved • Observation in usability testing • Video and interaction logs capture all the user operations • One-way mirrors or remote TV screen • Data is used to analyze what users are doing and provide insight into users’ affective reactions
Approaches to Observation • Observation in field studies • The observer may be a passive observer, participant observer or an ethnographer. • Goal is to cause as little disruption as possible. • Passive or outside observer – An observer in a class of boys and girls whose primary job to keep track of how much time a particular technology is used by each gender. • Participant observer – Participates in social conventions of a group, combines participation with maintenance of professional distance for unbiased observation. • Ethnography – Takes weeks, months or longer. Inside information of community activity is obtained.
How to observe:In controlled environment • Practical issues that need to thought of in advance: • Decide where users are located so that equipment can be setup in advance • Decide about modes of data capture e.g. video, interaction logs • Equipment testing to get expected performance (audio and video) • Get legal user consent • Prepare a script to guide users through the set of questions • It is important to make users feel comfortable • Problem with this approach: Observers do not know what users are thinking.
Controlled Environment – Think-Aloud Technique • The user is trying to perform a certain task. Observer wonders, what is going on, what is he thinking, why did he do that, etc. • Eternalize thought process. • A big problem is occurrences of silence during think-aloud process. • Maybe have two people work together and talk to each other.
Checklist of things to plan before going into the field • State the initial goal and questions clearly • Select a framework to guide activities in the field • Decide how to record events • Think about how to gain acceptance and trust of users under observation • Be prepared to go through these notes to weed out ambiguities • Highlight and separate personal opinion from what happens • Be prepared to refine and refocus the study for same or new user groups • Think about how to handle sensitive issues • Consider working as a team; conform notes with another member • Consider checking notes with an informant and plan to look at the situation from a different perspective
How to observe (contd.):In the field • Frameworks to guide observation • The person. Who? • The place. Where? • The thing. What? • The Goetz and LeCompte (1984) framework: • Who is present? • What is happening? • When does the activity occur? • Where is it happening? • Why is it happening? • How is the activity organized?
In the field • The Robinson (1993) framework • Space. What is the physical space like? • Actors. Who is involved? • Activities. What are they doing? • Objects. What objects are present? • Acts. What are individuals doing? • Events. What kind of event is it? • Goals. What do they to accomplish? • Feelings. What is the mood of the group and of individuals?
How to observe (contd.):Participant observer or ethnography • Checklist for doing ethnography • Identify the problem or goal and ask good questions to be answered by the study • The most important part of fieldwork is being there to observe, ask questions and record what is seen and heard • Collect variety of data. E.g. Notes, still pictures, audio and video • Be prepared to move backwards and forwards between the broad picture and specific questions • Analyze the data using a holistic approach in which observations are contextualized
Data Collection Techniques • Notes plus still camera • The least technical way of collecting data • Transcription the first step in data analysis • Photographs, sketches, etc. can be easily collected • Audio recording plus still camera • Less intrusive than video • More flexible and mobile • One drawback is transcribing the data which can be onerous if many hours have to be transcribed
Data Collection Techniques • Video • Advantage of both audio and video data; but more intrusive • Attention becomes focused on what is seen through the lens • Analysis of video can be long and time consuming • Sound may get muffled when recording in noisy conditions
Table source: Preece, Rogers, Sharp: Interaction Design: Beyond Human-Computer Interaction. Wiley Comparison
Table source: Preece, Rogers, Sharp: Interaction Design: Beyond Human-Computer Interaction. Wiley Comparison (contd.)
Indirect observation: tracking user activities • Diaries • Diaries provide a record of what users did, when they did it and what they thought about their interactions with technology • Useful when users and scattered and unreachable • Inexpensive and require no special equipment or expertise • They rely on participants being reliable and remembering to complete them • Participants usually remember incidents for better or for worse
Indirect observation: tracking user activities • Interaction logging • Includes analysing data from key presses, mouse movements etc. • Usually synchronized with video and audio logs • Time stamped to calculate how much time a user spends on a particular task • E.g. Explicit counters that record visits to a website. If you want to find out if adding a bulletin board increases the number of visits, comparison of traffic before and after is useful • Unobtrusive and large volumes of data can be collected
Ethical Concerns: They don't know we are watching. Shall we tell them? • If we tell users about logging they may react or change their behavior • It depends on the context, and how much personal information is collected • e.g. Companies tell about data logged for quality assurance purposes • Concerns arise when personal information is used to infer financial or health information
Analyzing, interpreting and presenting the data • Three main types of data are explored • Qualitativedata - interpreted & used to tell the ‘story’ about what was observed. • Qualitative data - categorized using techniques such as content analysis. • Quantitative data - collected from interaction & video logs. Presented as values, tables, charts, graphs and treated statistically.
Qualitative analysis to tell a story • Review the data after each observation to identify key themes • Record themes in a coherent yet flexible form • E.g. Laptops, audio recordings • Record the date and time of each data session • As themes emerge, check your understanding with informants or people you observe • Iterate until the story reflects observation • Report findings to a group in the form of a presentation or written report
Analyzing and reporting ethnographic data • Look for key events within a group that drive a group’s activity • Look for patterns or behavior among different players • Compare sources of data against each other • Report findings in a convincing and honest way • e.g. Software tools such as Ethnograph allow ethnographers to code their notes so that they can be searched, sorted and retrieved. The information can be printed out as a tree showing relationships of occurrences.
Qualitative analysis for categorization • Looking for incidents or patterns • Look for critical incidents, such as times when the user was obviously stuck. (marked by silence, looks of confusion) • E.g. Jurgen Koenemann-Belliveau et al(1994) used this approach to compare the efficacy of two versions of a Smalltalk programming manual for novice programmers. • They were able to identify specific problems by tracing through sequence of incidents and thereby achieving a holistic understanding.
Qualitative analysis for categorization • Analyzing data into categories • Content analysis provides a reliable and systematic way of coding content into meaningful sets of mutually exclusive categories • Categories should be orthogonal i.e. no overlap • Appropriate granularity must be selected • e.g. Training two people to use categories and letting them analyze the same data sample. If huge discrepancy in the analyses is observed, then either the training was insufficient or categories need to be redefined • Inter-research reliability rating- Percentage of agreement between two researchers, defined as ratio of number of items categorized in the same way to the total number of items observed
Qualitative analysis for categorization • Analyzing Discourse • Focus on the dialog, i.e. meaning of what was said rather than content • Discourse analysis is strongly interpretive • Language is viewed as a constructive tool and discourse analysis provides a way of focusing upon people’s use of language to construct versions of their worlds • E.g. Analyzing discourse on the Internet (e.g. in chat rooms, bulletin boards) has started to influence designers’ understanding about user needs in these environments
Quantitative data analysis and feedback • Quantitative data analysis • Video data collected is usually annotated to calculate performance times • The data stream from interaction logs is used to calculate performance times • Data is analyzed using means, standard deviations, t-tests etc. • Feeding the findings back into the design • Written reports make for easy reading and a good reference document • Include anecdotes, pictures, quotations and video clips to bring study to life and stimulate interest • Verbal presentations can be a powerful mode of feedback
Summary • Observation in usability study tends to be objective • In contrast, in participant observation the evaluator works with the user to understand their activities and beliefs • Ethnography uses a technique wherein the ethnographer immerse themselves in the culture they study • Observational data collection depends on the paradigm used • Combination of video, audio, diaries and logs can be used to collect observational data • Analyzing video and data logs can be tedious due to the sheer volume of data • Evaluators flag events in real time and return to examine them later. • Identifying key events is an effective approach
