Evaluation Techniques

1. Evaluation Techniques

2. Overview • Evaluation • tests usability and functionality of system • occurs in laboratory, field and/or in collaboration with users • evaluates both design and implementation • Evaluation should be considered at all stages in the design life cycle

3. Goals of Evaluation • Assess extent of system functionality • Assess effect of interface on user • Identify specific problems

4. Styles of Evaluation • Laboratory studies • Advantages: • specialist equipment available • uninterrupted environment • Disadvantages: • lack ofcontext • difficult to observe several users cooperating • Appropriate • if system location is dangerous or impractical • for constrained single user systems • to allow controlled manipulation of use

5. Styles of Evaluation • Field Studies • Advantages: • natural environment • context retained (though observation may alter it) • longitudinal studies possible • Disadvantages: • distractions • noise • Appropriate • where context is crucial • for longitudinal studies

6. Styles of Evaluation • Participatory Design • User is an active member of the design team. • Characteristics • context and work oriented rather than system oriented • collaborative • iterative

7. Evaluating Designs • Cognitive Walkthrough • Heuristic Evaluation • Review-based evaluation • Model-based evaluation

8. Cognitive Walkthrough • Proposed by Polson et al. • Evaluates design on how well it supports user in learning task • usually performed by expert in cognitive psychology • expert `walks though' design to identify potential problems using psychological principles • forms used to guide analysis

9. Cognitive Walkthrough • To perform a walkthrough you need four things • 1. A description of the prototype of the system • 2. A description of the task the user is to perform on the system. • 3. A complete written list of the actions needed to complete the task with the given prototype • 4. An indication of who the users are and what kind of experience and knowledge the evaluators can assume about them.

10. Cognitive Walkthrough • Given this information step through the action sequence ( number 3) • For each action answer the following questions • 1. Will the users be trying to produce whatever effect the action has? • 2. Will users be able to notice that the correct action is available? • 3. Once users find the correct action at the interface, will they know that it is the right one for the effect that they are trying to produce? • 4. After the action is taken will users understand the feedback that they are given?

11. Cognitive Walkthrough • For each taskwalkthrough considers • what impact will interaction have on user? • what cognitive processes are required? • what learning problems may occur? • Analysis focuses on goals and knowledge: does the design lead the user to generate the correct goals? • An example is expanded in Section 11.4.1

12. Heuristic Evaluation • Proposed by Nielsen and Molich. (www.useit.com) • Usability criteria (heuristics) are identified • design examined by experts to see if these are violated • Example heuristics • system behaviour is predictable • system behaviour is consistent • feedback is provided • Heuristic evaluation `debugs' design

13. Heuristic Evaluation • Example criteria - original • Simple and natural dialog • Speak the users language • Minimise user memory load • Be consistent • Provide feedback • Provide clearly marked exits • Provide shortcuts • Good error messages • Prevent errors

14. Heuristic Evaluation • Example criteria -improved • Visibility of system status • Match between system and real world • User control and freedom • Consistency and standards • Error prevention • Recognition rather than recall • Flexibility and efficiency of use • Aesthetic and minimalist design • Help users recognise, diagnose and recover from errors • Help and documentation

15. Review-based evaluation • Results reported in the literature are used to support or refute parts of design. • Care is needed to ensure results are transferable to new design.

16. Model-based evaluation • Cognitive models are used to filter design options. E.g. GOMS prediction of user performance. • Design rationale can also provide useful information in evaluating designs

17. Evaluating Implementations • Requires an artefact - simulation, prototype, full implementation. • Experimental evaluation • controlled evaluation of specific aspects of interactive behaviour • evaluator chooses hypothesis to be tested • a number of experimental conditions are considered which differ only in the value of some controlled variable • changes in behavioural measure are attributed to different conditions

18. Evaluating Implementations • Experimental factors • Subjects • representative • sufficient sample • Variables • independent variable (IV) • characteristic changed to produce different conditions. E.g. interface style, number of menu items. • dependent variable (DV) • characteristics measured in the experiment. E.g. time taken, number of errors

19. Evaluating Implementations • Hypothesis • prediction of outcome framed in terms of IV and DV • null hypothesis: states no difference between conditions - aim is to disprove this • Experimental design • Within groups design • each subject performs experiment under each condition. Transfer of learning possible but less costly and less likely to suffer from user variation. • Between groups design • each subject performs under only one condition. No transfer of learning but more users required and variation can bias results

20. Analysis of data • Look at data • Save original data • Choice of statistical technique depends on • type of data • information required • Type of data • Discrete • finite number of values • Continuous • any value

21. Analysis of data • Types of test • parametric • assume normal distribution • robust • powerful • non-parametric • do not assume normal distribution • less powerful • more reliable • contingency table • classify data by discrete attributes and count number of data items in each group

22. Analysis of data • What information is required? • is there a difference? • how big is the difference? • how accurate is the estimate? • Parametric and non-parametric tests address mainly the first of these. • Worked examples of data analysis are given in Section 11.5.1. • Table 11.1 summarizes main tests and when they are used

23. Observational Methods • Think Aloud • user observed performing task • user asked to describe what he is doing and why, what he thinks is happening etc. • Advantages • simplicity |requires little expertise • can provide useful insight • can show how system is actually used • Disadvantages • subjective • selective • act of describing may alter task performance

24. Observational Methods • Cooperative evaluation - variation on think aloud • user collaborates in evaluation • both user and evaluator can ask each other questions throughout • Additional advantages • less constrained and easier to use • user is encouraged to criticize system • clarification possible

25. Observational Methods • Protocol analysis methods • paper and pencil • cheap • limited to writing speed • audio • good for think aloud • difficult to match with other protocols • video • accurate and realistic • needs special equipment • obtrusive

26. Observational Methods • More protocol analysis methods • computer logging • automatic and unobtrusive • large amounts of data difficult to analyse • user notebooks • coarse level and subjective • useful insights • good for longitudinal studies

27. Observational Methods • Mixed use in practice. • Transcription of audio and video difficult and requires skill. • Some automatic support tools available • EVA • Workplace project

28. Observational Methods • Post task walkthrough • user reflects on action after the event • used to fill in intention • Advantages • analyst has time to focus on relevant incidents • avoid excessive interruption of task • Disadvantages • lack of freshness • may be post-hoc interpretation of events

29. Query Techniques • Informal and subjective • Cheap • Interviews • Analyst questions user on one to one basis, • usually based on prepared questions. • Advantages • can be varied to suit context • issues can be explored more fully • can elicit user views and identify unanticipated problems • Disadvantages • very subjective • time consuming

30. Query Techniques • Questionnaires • Set of fixed questions given to users. • Advantages • quick and reaches large user group • can be analysed more rigorously • Disadvantages • less flexible • less probing • Need careful design • what information is required? • how are answers to be analyzed?

31. Query Techniques • Questionnaires continued • Styles of question • general • open-ended • scalar • multi-choice • ranked

32. Choosing an Evaluation Method • Factors to consider (see also Tables 11.3-11.5) • when in cycle is evaluation carried out? • design vs implementation • what style of evaluation is required? • laboratory vs field • how objective should the technique be? • subjective vs objective • what type of measures are required? • qualitative vs quantitative • what level of information is required? • High level vs low level • what level of interference? • obtrusive vs unobtrusive • what resources are available? • time, subjects, equipment, expertise

33. Choosing an Evaluation Method • Tables 11.3{11.5 rate each techniques along these criteria.