Extraordinary Claims

1. Extraordinary Claims

2. “A wise man… proportions his belief to the evidence.” – David Hume

3. “Extraordinary claims require extraordinary evidence.” – Carl Sagan

4. Base rate neglect

5. Outcomes

6. Base Rates The base rate for any condition is simply the proportion of people who have the condition. Base rate of dog owners in Hong Kong: # of Dog Owners in Hong Kong ÷ # of Hong Kongers

7. Base Rate

8. As the base rate decreases, the number of true positives decreases.

9. Base Rate

10. Importance of Base Rates Therefore, the proportion of false positives out of total positives increases: False Positive ÷ False Positive + True Positive

11. Importance of Base Rates And the proportion of true positives out of total positives decreases: True Positive ÷ False Positive + True Positive

12. What Does It Mean? It means that even very good tests (low false positive rate, low false negative rate) cannot reliably detect conditions with low base rates.

13. Example Suppose the police have a test for telling whether someone is driving drunk: • If you are drunk, the test returns “positive” 100% of the time. • If you are not drunk, the test returns “positive” 95% of the time. The police randomly stop 1,000 drivers and test them.

14. The base rate of drunk drivers is: # of drunk drivers ÷ # of drunk drivers + # of sober drivers

15. High Base Rate The base rate of drunk drivers is: # of drunk drivers =500 ÷ # of drunk drivers =500 + # of sober drivers =500 Base rate = 50%

16. Outcomes

17. Outcomes

18. Outcomes

19. Good Test! Now we have 500 true positives and only 25 false positives: False Positive ÷ False Positive + True Positive

20. Good Test! Now we have 500 true positives and only 25 false positives: 25 ÷ 25 + 500 5% =

21. Low Base Rate The base rate of drunk drivers is: # of drunk drivers =5 ÷ # of drunk drivers =5 + # of sober drivers =995 Base rate = 0.5%

22. Outcomes

23. Outcomes

24. Outcomes

25. Bad Test! Now we have 5 true positives and 50 false positives: False Positive ÷ False Positive + True Positive

26. Bad Test! Now we have 5 true positives and 50 false positives: 50 ÷ 50 + 5 91% =

27. We saw this same phenomenon when we learned that most published scientific research is false.

29. Low base rate of truths.

30. High proportion of false positives.

31. Base Rate Neglect Fallacy The base rate neglect fallacy is when we ignore the base rate. The base rate is very low, so our tests are very unreliable… but we still trust the tests.

32. It Matters The Hong Kong police stop people 2 million times every year. Only 22,500 of these cases (1%) are found to be offenses. Even if the police have a very low false positive rate it’s clear that the base rate of offenses does not warrant these stops.

33. Prior probabilities

34. Probabilities A base rate is a rate (a frequency). # of X’s out of # of Y’s. # of drunk drivers out of # of total drivers. Sometimes a rate or a frequency doesn’t make sense. Some things only happen once, like an election. So here we talk about probabilities instead of rates.

35. Probabilities For things that do happen a lot the probabilities are (or approximate) the frequencies: Probability of random person being stopped by police = # of people stopped by police ÷ total # of people

36. Things that Only Happened Once • The 2012 Chief Executive Election • The 1997 Handover • The great plague of Hong Kong • The second world war • The extinction of the dinosaurs • The big bang

37. Evidence Drunk tests are a type of evidence. Testing positive raises the probability that you are drunk. But how much does it raise the probability? That depends on the base rate.

38. Evidence You can also have evidence for or against one-time events. For example, you might have a footprint at the scene of a crime. This raises the probability that certain people are guilty. How much? That depends on the prior probability.

39. Priors and Posteriors Prior probability = probability that something is true before you look at the new evidence. Posterior probability = probability that something is true after you look at the new evidence.

40. Relativity Priors and posteriors are relative to evidence. Once we look at new evidence, our posteriors become our new priors when we consider the next evidence.

41. Bayes’ Theorem P (data/ hyp.) x P(hyp.) ÷ P(data) Prior Posterior P(hypothesis/ data) =

42. Prior/ Posterior Positively Correlated P(hypothesis/ data) ∝ [P(data/ hyp.) x P(hyp.)]

43. What Does It Mean If the prior probability of some event is very low, then even very good evidence for that event does not significantly increase its probability.

44. “Extraordinary claims require extraordinary evidence.” – Carl Sagan

45. Hume on miracles

46. “A wise man… proportions his belief to the evidence.” – David Hume

47. Novel Testimony Suppose that we get testimony concerning something we have never experienced. Hume imagines someone from the equatorial regions being told about frost, and snow, and ice. They have never experienced anything like that before.

48. It’s Strange!

49. Hume thinks this person would have reason to disbelieve stories about a white powder that fell from the sky, covered everything by several inches, and then turned to water and went away.

50. It’s not that they should believe the stories are nottrue, just that they don’t have to believe they aretrue. We need more evidence, because the prior is so low.