Potential Pitfalls of Relying on Anecdotal Evidence

1. Potential Pitfalls of Relying on Anecdotal Evidence Jacqueline Pistorello, Ph.D. Catherine Choi Pearson, Ph.D. Assessment Implementation Team Student Services February 6, 2004

2. Workshop Orientation Objectives. At the end of this workshop, you will: Know at least three biases that may operate when we rely solely on anecdotal evidence Know at least three biases and fallacies that may occur even after we have access to empirical data Be able to list at least two potentially corrective actions against these biases Overall goal: Raise awareness Format: Interactive and participatory

9. What are the odds of that? Odds of dying in a car wreck: 1 in 18,585 Odds of dying in a plane crash: 1 in 354,319 Odds of dying by drowning in pool: 1 in 485,549 Odds of dying by earthquake: 1 in 7,865,886 Odds of dying by venomous spider bite: 1 in 55,061,200 Odds for the year 2000 from the National Safety Council Odds for the year 2000 from the National Safety Council

10. Car vs. Plane: Why? People tend to rely on information that is most salient in their minds Anecdotal evidence becomes important. End up paying attention to factors, like emotional reactions, instead of on the data.

11. Heuristics Defined Gray (1991) defines it as �any rule that allows one to reduce the number of operations that are tried in solving a problem� (p. 393). Heuristics are shortcuts. Rules of thumb based on experience (Federal Aviation Administration, 2004) Work well most of the time, but occasionally can lead to undesirable outcomes.

12. Three types of Heuristics (Prasad, 2003) Availability How easily instances or occurrences can be brought to mind Representativeness Assess the likelihood of occurrence of an event based on experiences with occurrences of similar events before Anchoring and Adjustment Starting at initial value and adjusting it to reach a final decision Bias in initial hypothesis that doesn�t easily shift to an alternative

13. Individuals Bill Clinton Dorothy Miller John F. Kennedy Maria Brown Theodore Roosevelt Harry S. Truman Sharon Smith Gloria Black Theresa Smith Dwight D. Eisenhower Laura Potter Ruth Ingram John Lilley Gerald Ford Ruth White Amy Jones

14. Availability Heuristic Are there more words with letter �K� as first letter or third letter? Are women more likely to be assaulted by strangers or friends? List six examples of when you were assertive. How assertive are you? (Schwarz et al., 1991).

15. Representativeness Heuristic Assess the likelihood of occurrence of an event based on experiences with occurrences of similar events before Never won a radio contest before, so this time she is sure she will win. Won�t get merit this year because received it for the last 5 years.

16. Representativeness 8 coins are flipped How many will turn up heads? In reality, there are 256 possible outcomes for those 8 coins. Only 70 of them have 4 heads. 70/256 = 27% chance of 4 heads.

17. Going back to the list� What names do you remember? How many men on the list? How many women on the list? 7 men 9 women7 men 9 women

18. Exercise Half of the room close your eyes Other half will see a numerical expression on the screen Estimate the product, write it down without saying anything Will repeat with the other half of the room

20. 8 x 7 x 6 x 5 x 4 x 3 x 2 x 1

22. 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8

24. Estimating product First half of the room Second half of the room Study found Starting with 8 x 7 = 2,250 Starting with 1 x 2 = 512 Correct answer for both = 40, 320 Tversky & Kahneman, Science, 1974

25. Anchoring and Adjustment(Tversky & Kahneman, 1974) Different starting point yields different estimates, which are biased towards the initial value. Insufficient adjustment Study on estimating number of African countries Starting with 10 = 25; starting with 65 = 45 Example: Estimating salary for a new employee Base it on past employee�s salary and adjust from there. Regardless of changes to the job or new requirements or demands.

26. Another aspect of anchoring Bias in initial hypothesis that doesn�t easily shift to an alternative Standford study on capital punishment as deterrent of murder or not Whatever study supported their initial position was viewed as �more convincing� and �better designed� Rule-governed vs. contingency-shaped behavior

27. Student Days at the University of Nevada, Reno International graduate student, Raoul was working tirelessly in his lab and living in University-owned graduate student housing Las Vegas freshman, Tony, was taking advantage of on-line services but struggling to afford living in the residence halls Student leader Marsha was enjoying the new student union�s multicultural center Prospective student, Mya, was dreaming of attending the university

28. In the meantime, in a rival institution� John, whose GPA borders on 1.5, was selling fake IDs from his technologically challenged dorm room to underage students. Despite the ineptitude of their residence hall staff and police, John was arrested and charged with the crime. He was sentenced to 5 years in prison He has served 1 year and is now up for parole.

29. Discussion You are the parole board: What questions would you ask him in making your decision? What kind of data would you consider most relevant? What heuristics could apply in this situation?

30. Parole decisions Parole commissioners use �instinct� or �gut� to make determinations regarding parole. Ask questions about psychological history, searching for telling detail, clue to convince them that the prisoner is no longer a threat to society. Tom Miller �you look in their eyes, you can feel, you know, if they are being sincere or not. And you learn to see right through them.�

31. Porteus maze Trace maze with pencil. Pencil lifts predicts impulsive behavior. Better predictor of recidivism than parole boards. But, probably still would not be listed as one of the 10 best predictors of violence by psychologists or psychiatrists.

32. Meehl and Faust Accuracy rates very low in predicting recidivism Work of Meehl and Faust on calibrated statistical formulas to predict behaviors. Shown that numbers consistently beat out intuition in decision making.

33. Mark Twain Get your facts first and then you can distort �em as much as you please.

34. Now you have the data:What to watch out for (Meehl, 1974) Biased recollection and interpretation of data Buddy-buddy syndrome All evidence is equally good Reward everything- gold & garbage alike Feeble inferences Shift in evidential standards Ignoring statistical logic Recognizing there is difference between statistical and practical significance

35. Some common fallacies (Meehl, 1974) Sick-sick fallacy �Me too� fallacy Uncle George�s pancakes fallacy Understanding makes it acceptable, does not require change Hidden decisions Deceiving ourselves b/c might be challenged

36. So, what are we trying to say? Not that every decision needs to be data-based However, data make(s) us less susceptible to heuristics It is wise to be aware that we are subject to many biases AND, that we often are not aware of our own biases This does not only apply to laypeople

37. What can we do about it? Collect objective data as much as possible Beware of heuristics and own biases Create a community where disagreement and challenges are encouraged Always entertain multiple hypotheses What are we not seeing? Kyoto Garden Focus on effectiveness

39. References Meehl, P.E. (1974). Why I Do Not Attend Case Conferences. Psychodiagnosis: Selected Papers (pp. 225-302). Minneapolis: University of Minnesota Press. Tversky, A. & Kahneman, D. (1974). Judgment Under Uncertainty: Heuristics and biases. Science, 185, 1124-1131.