Trick or treatment? Evaluating the quality of structured risk management decisions

1. Trick or treatment?Evaluating the quality of structured risk management decisions Joe Arvai The Ohio State University Decision Research

2. Outline Structured DM in brief A practical example Questions of quality (3 experiments) Parting thoughts

3. SDM In Brief • Structured decision approaches have their roots in… • …the observation that people tend to have tremendous difficulty with making decisions that involve multiple objectives and, therefore, tradeoffs.

4. SDM In Brief • Structured decision approaches have their roots in… • …studies of the constructive nature of preferences in response to available cues.

5. SDM In Brief • Structured decision approaches are designed based on… • …“value focused thinking.”

6. SDM In Brief • Structured decision approaches are designed based on… • …the literature dealing with normative decision making, specifically the steps required for a complete analysis of a given decision

7. SDM In Brief • Structured decision approaches are designed based on… • …methods for decreasing the cognitive burden associated with complex choices.

8. The Case of Water Use Planningin British Columbia • Work with B.C. Hydro on a comprehensive, stakeholder-based development of revised operating plans at all major hydroelectric facilities. • In response to increasing, competing demands on provincial water resources

9. Scale of Study

10. The Case of Water Use Planningin British Columbia • Multiple Objectives: • Electricity generation/trade • Environmental quality • Water, land, & air • Recreation opportunities • Cultural values • Learn over time; reduce uncertainty http://www.bchydro.com/wup/

11. Water Use Planning in B.C.The Basic Steps • The approach is based on work with stakeholders on: • Eliciting objectives from various stakeholders. • Identifying a series of workable options for management. • Establishing attributes of/measures for each objective. • Generating a matrix across these objectives and options and addressing the tradeoffs that selecting one option over another entails.

12. Water Use Planning in B.C.Participants • BC Hydro (Crown Corporation) • Federal Government • Provincial Government • Local Government • First Nations • Community Stakeholders • Home owners, business operators, etc.

13. Water Use Planning in B.C.Objectives, Attributes, Measures

14. Water Use Planning in B.C.e.g., Financial Objectives & Attributes

15. Water Use Planning in B.C.e.g., RecreationObjectives & Attributes

16. Water Use Planning in B.C.Assessing Value

17. 1.0 0.8 0.6 0.4 0.2 RISK PROFILES Nat Sum Probability Range of Values Nat 60 40 100 Sum 80 10 140 140 20 60 100 Expected Low High Expected Value Water Use Planning in B.C.Risk & Uncertainty

18. Water Use Planning in B.C.

19. Water Use Planning in B.C.

20. A High Quality Process? • Government Support • Approach originally conceived as a pilot project at a single site • Now implemented at 23 sites province wide (18 Water Use Plans completed to date) • Cost-effective • Original budget to complete all 23 plans = $25 Million • Revised budget = Approx. $14 Million • Participant Satisfaction • Historically adversarial groups work together • BC Hydro has a clear mandate • Consensus plans often achieved

21. Experiment 1Quantitative Measures of Quality • Designed to compare two approaches for involving stakeholders in water use planning in B.C… • …small groups (7-10) • …both conditions provided with the same information • …“Structured” vs. “Unstructured” process

22. UNSTRUCTURED 1. Self-rating questions 2. Technical Information newspaper article informative booklet audio-documentary 3. Group Discussion 4. Evaluation of policy alternatives referendum willingness to pay 5. Self-rating questions STRUCTURED 1. Self-rating questions 2. Technical Information newspaper article informative booklet audio-documentary 3. Decision structuring characterize values & objectives group discussion about values and objectives. objectives ranking/tradeoffs 4. Evaluation of policy alternatives referendum willingness to pay 5. Self-rating questions Experiment 1Quantitative Measures of Quality

23. Experiment 1Results

24. Experiment 1Results F

25. Experiment 1Conclusions • Based on participants’ self-ratings and an analysis of deliberation periods, we conclude that the the structured approach leads to higher quality decisions. • Arvai, J. L., R. Gregory, and T. McDaniels. 2001. Testing a structured decision approach: Value-focused thinking for deliberative risk communication. Risk Analysis, 21: 1065-1076.

26. Experiment 2Analyzing Choices • Decision structuring to alleviate embedding in environmental valuation • …when a good is assigned a higher value on its own vs. when it’s part of a more inclusive set. • e.g., 5:1 differences in WTP for a single vs. a set of disaster preparedness services

27. Experiment 2Analyzing Choices • Providing much-needed structure may help to overcome the embedding problem by helping decision makers to think about the components of a valuation problem.

28. VERSION A 1. Technical Information newspaper article informative booklet audio-documentary 2. Decision structuring characterize values & objectives group discussion about values and objectives. objectives ranking/tradeoffs 3. WTP 1 Rivers 4. WTP 10 River VERSION B 1. Technical Information newspaper article informative booklet audio-documentary 2. Decision structuring characterize values & objectives group discussion about values and objectives. objectives ranking/tradeoffs 3. WTP 10 Rivers 4. WTP 1 River Experiment 2Analyzing Choices

29. Experiment 2Analyzing Choices No Embedding W/X = Y/Z 0.42  0.25 W > Y Z<X

30. Experiment 2Conclusions • Based on participants’ mean WTP judgments, embedding was not alleviated (according to the ratio standard). • McDaniels, T., R. Gregory, J. L. Arvai, and R. Chuenpagdee. 2003. Decision structuring as a means of alleviating embedding in environmental valuation. Ecological Economics, 44: 33-46.

31. Experiment 32-Part Evaluation • An experiment that would measure both self-ratings of quality and subjects’ choices…

32. UNSTRUCTURED 1. Self-rating questions 2. Technical Information 3. Choice Task Funding allocations across three risk problems 4. Self-rating questions STRUCTURED 1. Self-rating questions 2. Technical Information 3. Decision Structuring Objectives ranking, tradeoffs Linking objectives with management problems 4. Choice Task Funding allocations across three risk problems 5. Self-rating questions Experiment 32-Part Evaluation

33. Experiment 3Three Risk Problems

34. Experiment 3Self-Rating Results

35. 1=Overpopulation 2=Disease 3=Trails Experiment 3Ranked Objectives

36. Experiment 3Funding Allocations

37. Experiment 3Conclusions • Appears to be a disconnect between self-ratings of quality and actual funding choices • Wilson, R.S. and J. L. Arvai. 2004. Evaluating the quality of structured risk management decisions. In Review.

38. Parting Thoughts • Level of facilitation, time for deliberation, increased attention to tradeoffs, etc. all seem to be critical.

39. Parting Thoughts • Anecdotal observations or evaluations based on self-reports alone are likely insufficient for evaluating the quality of structured decision approaches.

40. Parting Thoughts • Affective responses to stimuli exert powerful influences on risk judgments

41. Parting Thoughts • Not suggesting that the outcomes of all structured decision making approaches are suspect.

42. Thanks Tim McDaniels Robin Gregory Ying Chuenpagdee Robyn Wilson Louie Rivers Dan Ohlsen SSHRC NSF OARDC EPI