Evaluation Methods

Evaluation Methods April 20, 2005 Tara Matthews CS 160

In 160 We’ve Covered… • Task Analysis & Contextual Inquiry • Cognitive Walkthrough • Heuristic Evaluation • WOZ usability study w/ paper prototypes

There are many more methods… • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

How to chose a method? • Stage of study • formative, iterative, summative • Pros & cons • Metrics • depends on what you want to measure • Qualitative vs. quantitative • Research perspective • CS vs. psychology vs. sociology

Pros & Cons • Realism • Precision • Generalizability • Time & cost • Researcher expertise

Methods • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

Survey • Online / paper questionnaires distributed to target audience • Can be used to • tabulate quantitative data • gather qualitative feedback (opinions, feelings, etc.) • Useful at any time in study

Survey • Pros • Easy to get a large number of responses. • Quick and easy to conduct. • Highly generalizable. • Cons • Self-selection. • Participants often only offer enough information to answer the question. • Can miss details. • Low in realism and precision.

Interview • Evaluators formulate questions on the issues of interest. • Interview representative users, asking them these questions in order to gather information desired. • Interviewer reads questions to user, who replies verbally; interviewer records responses.

Interview • Pros • Quick and easy to conduct. • Gives designer quick feedback on a range of ideas. • Can get a person’s initial reaction to an idea. • Can get detailed information from a person. • Cons • Often takes place away from natural setting. • Question wording or interviewer “body language” can bias answers. • High probability of false positives and missed problems (e.g., users may not have a clear idea of how an app will be used). • Can miss details if interviewer doesn’t know what issues to draw out.

Controlled Lab Experiment • In lab, manipulate one feature of a system to assess the causal effects of the difference in that manipulated feature on other behaviors of the system. • Example: • in lab, show users 4 versions of a website: • blue, yellow, red, and black text • measure time to find specific words • compare

Controlled Lab Experiment • Pros • Provides precise, quantifiable data. • Easier to draw inferences from data. • Relatively quick. • Can get a medium-sized number of participants. • Cons • Short duration of a lab experiment may not be enough to allow users to become accustomed to an app. • Not a natural setting – interaction may not be normal.

In-lab Observation • Participants come to lab to "use" an interface • Given sample tasks to complete with it • Evaluators observe and possibly audio- or videotape • Participants may "think out loud" • Can use lo-fi prototype (for a project in the design stage) to an almost-complete interface • Evaluators note participants’ • emotions, exclamations, facial expressions, and other "qualitative" data • take note of quantitative data such as time to complete a task or number of errors

In-lab Observation • Pros • Relatively quick. • Can get a medium-sized number of participants. • Cons • Observations are subjective and error prone. • Short duration of lab observation is not enough time for user to get accustomed to using the interface. • Not a natural setting – interaction may not be normal.

Controlled Field Experiment • In natural setting, manipulate one feature of a system to assess the causal effects of the difference in that manipulated feature on other behaviors of the system. • Example: • Participants use 3 different input devices in their own office: mouse with 1, 2, or 3 buttons • Perform a set of tasks • Measure differences

Controlled Field Experiment • Pros • Less intrusive than most other evaluation methods. • Provides more precise data than field observation. • Can observe natural behavior of user (though some part of the system will be controlled/unnatural). • Cons • More intrusive than field observation. • Less natural than field observation.

Field Observation Study • Evaluator makes direct observations of “natural” systems • Takes care to not intrude on / disturb those systems • A.K.A. “ethnography”

Field Observation Study • Pros • Only way to observe natural behavior of user & interaction between user & tools. • Cons • Difficult and time consuming. • Hard to get permission to observe people. • Observations are subjective and error prone. • Cannot make strong interpretations from observations. • Not very generalizable.

Heuristic Evaluation • Pros • Quick and easy. • Cons • Nielson’s heuristics may not be as relevant to non-GUIs. • Results in false positives in missed problems, especially when experts are not part of target audience.

Cognitive Walkthrough • Pros • Quick and easy. • Cons • Results in false positives and missed problems when evaluator is different from target audience.

Automate Observation Study • Techniques include • video or audio recording of user • pop-up screens • screen shots • time logging • log users actions (collecting statistics about detailed system use)

Automate Observation Study • Pros • Eases burden on observers for data collection & analysis. • Cons • Setup is often more time-consuming to complete. • Harder to get approved if it involves analysis of videotape or audiotape. • May miss nuanced/interpretive details.

Experimental Simulation • In-lab experiment that is as much like some real situation as possible. • Example: • ground-based flight simulator • behaves as closely as possible to a real flight • still under researcher control

Experimental Simulation • Pros • Still fairly precise. • More realistic than in-lab experiment. • Cons (same as lab exp.) • Short duration of a lab experiment may not be enough to allow users to become accustomed to an app. • Not a natural setting – interaction may not be normal.

Claims Analysis • Claim = statement that a certain aspect (button, scrollbar) of a design has psychological implications reflected in how capable a user is in using that design • UI artifacts are listed along with their design features & pros/cons • Helps • select among alternative designs • clarify questions to be analyzed through user testing by stating how the design should work (in claims)

GOMS • A method to describe user tasks and how a user performs those tasks with a specific interface design • Views humans as information processors • Small number of cognitive, perceptual, and motor operators characterize user behavior • To apply GOMS: • Analyze task to identify user goals (hierarchical) • Identify operators to achieve goals • Sum operator times to predict performance • GOMS = • Goals: What a user wants to accomplish • Operators: Cognitive or physical actions that change the state of the user or the system • Methods: Groups of goals and operators • Selection rules: Determine which method to apply

GOMS • Pros • Predict human performance before committing to a specific design in code or running user studies • Many studies have validated the model (it works) • Cons • Assumes error-free, skilled user behavior • No formal recipe for how to perform analysis • Significant time investment

Computer Simulation • Creating a complete & closed system that models the operation of the concrete system without users. • Example: • geophysical process going on in connection with the eruption of Mount St. Helens

Computer Simulation • Pros • Supposedly high in realism (depends on accuracy of data/system replication) • Cons • Low in precision & generalizability

Formal Theory • Formulating general relations (propositions, hypothesis, or postulates) among a number of variables of interest. • Pros • Relatively generalizable • Cons • Not realistic or precise

How to chose a method? • Stage of study • Pros & cons • Realism • Precision • Generalizability • Time & cost • Researcher expertise • Metrics • Qualitative vs. quantitative • Research perspective

Methods • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

Early Stage • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

Iterative & Summative Stages • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

Realism • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

Precision • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

Generalizability • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

Time & Cost • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

Researcher Perspective • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

Metrics: examples • Traditional GUIs: • efficiency (time to complete task) • accuracy (# of errors) • simplicity • Peripheral Displays: • awareness (recall) • distraction (dual-task behavior) • aesthetics

Peripheral Displays • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

Questions?

Evaluation Methods