Human-Computer Decision Making: The View from Psychology

Human-Computer Decision Making: The View from Psychology Earl Hunt University of Washington

Acknowledgments • Collaborators on research project • Susan Joslyn UW Psychology • Karla Schweitzer UW Psychology • David Jones UW Applied Physics Laboratory • General inspiration and ideas-writings and talks. • Lee Beach U. of Arizona (formerly UW) • Gary Klein Klein Associates • Steven Poltrock Boeing Corporation • Steven Hunt Unicru Inc.

Support of Research, Preparation of Paper • Department of Defense Multidisciplinary Research Initiative • Office of Naval Research Grant N00014-01-10745 to the University of Washington • Adrian Raftery (Prof. of Statistics, UW) Principal Investigator

Possible Mixed Mode Decision Making • Computers tell humans what to do • Rental car check in. • Not relevant • Humans tell computer what to do • Some aspects of fly by wire • Not relevant • Computers advise and gather data, humans decide • Common • That’s what this is about.

Point of Talk • There is always an interface • Computer system designers must consider • Cognitive psychological aspects, not just human engineering • Social context of decision making situation

Plan of Talk • Psychological theories of decision making • Two studies of computer-human decision making • Heath and Luff study of British medical records system • Research of my colleagues and myself on Naval Aviation weather forecasting • Concluding comments.

Models of Decision Making

Von Neumann-Morgenstern Model • Life is a lottery: • Evaluate the rewards (utility) • Factor in the probabilities • Choose action with highest expected value • This is often seen as the normative model • Ideal for computerization!

Objections • Kahneman and Tversky (modifications) • People are poor estimators of probability • Subjective scale needed • May not be a probability calculus • Prospect theory-modify utility scale • Kahneman and Tversky (basic objection) • Framing. Decision situation changes if you focus on profit or loss • Merchants advertise discounts for cash. How many advertise charges for credit?

Radical Alterations • Richard Wagner: Rejection of lottery model • LeeRoy Beach: Image Theory • People make decisions based upon goals and their view of permissible ways to attain them • Example: University cheating • Progress by scenario creation and monitoring • Gary Klein: Pattern driven decision making • Decision making is largely forward driven • Depends upon recognizing situation, taking action. • Example: Fire commander’s action as fire progresses.

Implications • Von Neumann-Morgenstern approach good for computer-alone decision making • Example: programmed stock trading. • Does not fit into human thought process

Computers supporting human thought • Provide human with information in way human wants it. • Do not restrict human thought processes. • Don’t rely on applying mathematical decision theory except for autonomous computing

Medical Study • Ref: Heath, C. & Luff, P. (2000) Technology in action. Cambridge, Cambridge U. Press • Topic: Studied conversion of medical clinical records from written notes to computer forms • Forms consulted and/or filled out by the physician during the consultation.

Key points • Computer forms supported decision making by a high-level expert • Fully computerized systems proposed • We aren’ t there yet • Decision made • In social situation • Under considerable time pressure

Written Records • Flexibility • Spontaneously developed • Humans are trained to write standard forms • But, concept of defeasability • Most of the time standard forms • Any rule should have exceptions • Example (mine, not C and L) • Required: Case History, Diagnosis, Prognosis • Written: Drunk Again.

Computer Records • Standardized forms • Probably better for standard or anticipated issues: Made sure physician did not overlook something • Do not support defeasability • Or for special notes to other physicians • Interview was distorted • Physicians turned away from patient • Did not distract decision making about presented facts • Did distract physician-patient interchange, chance for spontaneous remarks by patient

Decision Making by Naval Weather Forecasters • Naval Aviation Forecast (Whidbey Island NAS, Whidbey Island, WA) • Writing Terminal Aerodrome Forecast (TAF) • Anticipated conditions at local airport • Similar TAF is filed for commercial airports by National Weather Service

Conditions of TAF • TAF written by PO1/c • Has experience as observer • Has gone to Navy METOC school. • Far less training that NWS meteorologist • (Navy does have more trained people at Fleet Weather Service) • TAF is written while other things are in process • Weather briefing for flights

information available to forecaster • Recent observations in region • Satellite observations • Numerical model predictions • More than one model is available • Models do not always agree • Different initializations • Different physics in model • One model can do best for awhile, then another • NO ONE could look at all of these variables!

Observations and formal experiments • Spent time observing how people used models • Forecasters would select “favorite” model, then adjust it from observations and satellite patterns • Did not spend much time on other models • Did not look at model history

Is the model doing a good job? Satellite Numerical Model: MM5 Evaluation involves comparing patterns (e.g. position of low) in numerical models and satellite image (actual weather).

Example: Forecaster D compared the observed pressure to what the model had predicted. Access current pressure 29.69 Calculate difference 29.69-29.64=.05between current and forecast (error) Access predicted pressure 29.57for forecast time Adjust predicted pressure 29.57-.05=29.52based on current error Adjust predicted pressure 29.52+.02=29.54based on model bias

How effective is this • Have compared model and TAF • General results, observers are quite good

.68 Numerical Model Human Forecast .56 .39 Actual Multiple R= .68 • Wind Speed From Winter, Spring 2003 • Correlation between Navy Forecasters’ prediction and the observed wind speed was greater than the correlation between numerical models and observed wind speed.

The variance in actual wind speed accounted for by the human forecast subsumed that accounted for by the numerical model. Observed=1 Numerical Model alone=.15 Human Forecast=.46

How general are these results? • Similar results have been obtained for other parameters (e.g. barometric pressure), other time periods. • Work rather difficult in summer! • Are continuing studies, for stormy period • Have observed similar results in artificial studies using UW undergrads, made up “cover story” • Select best recommendation • Adjust slightly, but actually pay little attention to non-favored recommenders.

What is being missed • Statistical studies show that the information in non-preferred models is valuable • But studies in literature are consistent with our findings • We do see multiple models in NWS • As did Hoffman and colleagues, studying NWS • Differences • Better trained individuals • Much less time pressure.

Probability Product: MM5 ACME Ensemble: Probability of winds greater than 20 knots

Stoplight Graphics Applied Physics Lab (UW) approach

We are studying simplified models, to see how much history checking is actually done

YOUR TASK: You need to make a forecast for atmospheric pressure for airline pilots. Pressure is used by the altimeter in determining the altitude of the plane. Pilots set the altimeter based on the predicted pressure. Pilots use altimeter readings to determine their altitude, which is especially important when landing the plane. If the pressure forecast is too high, the altimeter will read ground level too late and the plane may crash. It is important for your pressure forecast to be accurate, but you want to predict the lowest pressure, without going below, in order to safely land. You will make the forecast using computer model predictions alone. You have no other information. There are 3/7 models you may use. You may use as many or as few of them as you wish. The important thing is to make your best forecast. Clicking on the “prediction” button will give you information regarding each models’ prediction. Clicking on the “history” button will give you information regarding that forecasters previous 10 forecasts. Please make your best judgment of what the pressure will be. 1017 millibars Model A History Prediction Model B History Prediction Model C History Prediction Model D History Prediction Model E History Prediction Model F History Prediction Model G History Please type your forecast here: This chart will show trends in error- if the model is consistently high or low, large error or small error

Questions to consider in design • Peachy keen computer systems not enough • Consider context of use • Look especially at time pressures and social situations • How much flexibility should the human have? • Can system designer anticipate everything? • French bureaucrats n’cest posible !

Things that reduce flexibility • Situational factors • Time pressures, time sharing • Lack of training by individuals • Drowning people with information • The computer system itself • These are specific cases of ->

The curse of functionality • Added functionality may increase difficulty of use—actually reduces flexibility • Added probability information in weather forecasting seems to make little difference • MICROSOFT WORD, UNIX changes? • Research required on methods of increasing functionality and information for decision making without increasing complexity of information search

Take home message • These issues are research issues • Local decisions in specific situation • More general principles needed. • Focus groups and informal observations should be treated with suspicion • Random choice of users • Formal experimentation: statistical issues • Army MANPRINT guidance • Support your local cognitive psychologist

Human-Computer Decision Making: The View from Psychology