Measuring Human Performance

1. Measuring Human Performance Muckler(1992) Kantowitz(1992) Hennessy(1990)

2. Muckler(1992) Selecting Performance Measures: “Objective” versus “Subjective” Measurement

3. The Human Observer • “Alas, no fundamental step in science or technology is independent of the human” (Muckler 1992) • “The single most widely used measuring ‘instrument’ in science is the human observer” (Muckler 1992) • Why, then, do we try and remove the human from the equation?

4. Selecting Measures • Accoring to Muckler, there are no standard methodological algorithms for selecting a set of measures for any given situation • We tend to rely on methodologies from other disciplines • Most of these have a high focus on objective measures

5. Subjective vs. Objective • Muckler argues that we don’t really have any objective measures that don’t have some level of subjectivity to them • Expert ratings vs. self report • Selecting which statistical method for analysis • Interpreting the results

6. Subjective vs. Objective • Experimenters may influence their participants to introduce error • Orne(1962) found that participants will often attempt to be “good” subjects – pointless tasks • Even if the experimenter doesn’t influence the subject, this interaction has the potential to cause data distortion

7. Subjective vs. Objective • Subjective measures sometimes correlate with objective measures • Sleep studies – time falling asleep and while asleep • Work studies – task/time relationships • Combat stress reaction and post traumatic stress disorder • Why, then, would we want to ignore subjective measures?

8. Subjective vs. Objective • “Concluding that one measure is right and one wrong when the two disagree is to assume a simplicity that probably does not exist” (Muckler 1992)

9. Selecting Measures • Define what needs to be measured before looking at how to measure • Once a set of measurable items is selected, each item should be examined to determine the best way or ways to measure it

10. Selecting Measures

11. Kantowitz(1992) Selecting Measures for Human Factors Research

12. Measurement Fundamentals • Reliability • Consistency of a measure • Validity • Index of the truth of the measure • Usefulness • “Even if researchers are successful in meeting high standards of reliability and validity this does not guarantee that the human factors research will be useful.” (Kantowitz1992)

13. Subject Representativeness • Who should participate in research? • Only those who are members of the target population? • Peas, Animals, College students?

14. Variable Representativeness • Do the selected data sets and variables represent the population? • Kantowitz proposes we use the following to select representative variables: • Practical experience of researchers or subject matter experts • Theory and testability

15. Setting Representativeness • How well does the study setting match the target environment? • Physical realism vs. psychological realism • “Good subjects” problem – again

16. System Performance Measures • “It is seldom easy to obtain a measure that reflects overall system performance…Considered as a stand-alone system, the human operator is even more complex than a nuclear power plant.”

17. Theory-based Selection • All too often we select measures based on the experience of the the researcher • Theory-based selection provides a much more universal selection criterion. • Benefits of developing and using theory in selecting measures?

18. Theory-based Selection • Fill in data gaps • Predicts results prior to project instantiation • Prevents reinventing the wheel • Offers a normative basis for human behavior • The best practical tool – it can be used over and over for no cost

19. Theory-based Selection • Signal detection theory • Optimal control theory • Substantive information-processing theory • Psychometric theory

20. Theory-based Selection • Power plant example • First studying how heat stress affects attention • Develop a theory based on that study • Then apply that theory to the other tasks in the power plant

21. Hennessy Practical Human Performance Testing and Evaluation

22. Background • Military systems always undergo rigorous testing before being fielded • Prior to mid-1980’s these tests were wholly focused on the equipment • There are tests in place now to catch some usability issues, but only the major ones

23. Current Problem • “A good human performance measurement system has yet to be developed for meaningful tasks being performed in realistic environments.” (Hennessy 1990) • Two reasons for this: • Measurement is difficult • Most prevalent measures are not useful • Hennessey believes the root cause of this is too much emphasis on the importance of objective performance data

24. Measure Like Machines • There is a tendency to measure human performance the same as equipment performance • Others will look at the system output as a direct measure of the human performance

25. Measure Like Machines Machines People Carry out intentions Are unpredictable Have unknown design Indefinite output Many determinants Variable behavior Unstable over time Integrated • Perform functions • Are predictable • Have known design • Definite output • Few determinants • Repeatable Behavior • Stable over time • Discrete Parts

26. Lab to the Field • The lab environment is brought to the complex environment • Objective and subjective measures will be captured with an emphasis on the objective • Problems start when trying to actually come up with the tests to run and how to measure performance • Once testing is complete, often we find our assumptions have been incorrect

27. Lab to Field • Drawbacks of the objective data collected: • Missing data • Unknown factors • Inconclusive findings • Objective measures always include subjectivity

28. Lab to Field • “The lack of statistical reliability is offset with a statement to the effect that the subjective measures support the interpretation of the statistically significant objective data. Therefore, it is concluded, the inference drawn from the objective data is probably correct. In effect, the objective cart is placed in front of the subjective horse” (Hennessy 1990)

29. Subjective Measures • Observational data isn’t always subjective • Subjective measures often has less error

30. Subjective Measures • Economical in cost and time • Directly measure performance of interest • Context can be taken into account • Sensitivity of measurement • Results are available quickly • Performance of concurrent tasks can be distinguished • Cognitive tasks can be measured

31. Realistic Approach • Laboratory methods will not work in the field • Imperfect information is better than no information • Subjective measures are useful

32. Realistic Approach • Constructing a performance hierarchy • Obtaining aggregate weightings • Use of video recording for documentation and detailed analysis

33. Bonus Questions • What was the structural issue with the Muckler article? • What was stamped at the top of every page in the Kantowitz article? • The Hennessey article was found in a textbook – which chapter of the book contained this article?