How to Lie With Statistics

1. How to Lie With Statistics

3. I'm so confident! • The universe is not one of clockwork • Everything we measure has uncertainty • How can we confidently draw conclusions and make decisions without sacrificing our intelligence?

4. Quantify Your Uncertainty! • Family of intervals to handle this • Confidence • Prediction • Comprised of • Point estimator • Degree of Confidence • Uncertainty • Rely on • Underlying distribution (assumption) • Data collection

5. Formula Z*,t* Margin of Error

6. What could go wrong? • Poor data collection • "I talked to my 10 closest friends – Romney has got the election in the BAG!" • Wrong assumptions • Excessive uncertainty • Unrealistic expectations • Degree of confidence is too high/low • Wrong type of interval • Prediction versus Confidence

7. Degree of Confidence • What are we confident in? • The Process • If we collect data from the same population, over and over and over again, we can expect our interval to contain the true (correct) value x% of thetime where x is ourdegree of confidence(confidence level) • "The feeling or belief thatone can rely on someoneor something"-- What I got Googling"What is Confidence"

8. Degree of Confidence • What is confidence NOT? • A probability of any single interval containing the true value • The % of all population values contained in the interval • A product of your self-esteem

9. Lancet Survey • October 29, 20041The Lancet published a study on the effects of the US invasion of Iraq, specifically on the number of additional civilian deaths due to the invasion • The data estimated 98,000 excess deaths • They were 95% confident that the true number of excess deaths was between 8,000 and 194,000 1Les, R., Lafta, R., Garfield, R., Khudhairi, J., & Burnham, G. (2004). "Mortality before and after the 2003 invasion of Iraq: Cluster sample survey. The Lancet, 1857-1864.

10. Lancet Survey • October 11, 20062The Lancet published a follow-up study • The data now estimated 654,965 excess deaths • They were 95% confident that the true number of excess deaths was between 392,979 and 942,636 • Claimed that they asked for death certificates 87% of the time and that 92% of the time they asked they were shown a certificate (80% of reported deaths had a certificate to verify the death) • Iraqi Ministry of Health only issued ~50,000 death certificates for post-war violent deaths 2Burnham, G., Lafta, R., Doocy, S., & Roberts, L. (2006). Mortality after the 2003 invasion of Iraq: a cross-sectional cluster sample survey. The Lancet, Retrieved from http://www.brussellstribunal.org/pdf/lancet111006.pdf

11. Statistical Ties • Des Moines Register Iowa Poll Results Published 9/29/2012 • Obama 49% • Romney 45% • Other 4% • Undecided 2% • Margin of Error ±3.8% • Who is winning? • Register claims Obama has a 4% lead • Statistically inconclusive • The poll results give an ~84% chance that Obama is really ahead

12. Multiplicity of Error

13. Multiplicity of Error • Dr. Elisabeth Targ made headlines when she identified an effect of remote faith healing • The study results were published in the peer-reviewed journal The Western Journal of Medicine under the title "A randomized double-blind study of the effect of distant healing in a population with advanced AIDS. Report of a small scale study" • Results from the double blind study showed that members of the control group spent 600% more days in the hospital and contracted 300% as many AIDS-related illnesses • Considered the most rigorous, scientifically valid experiment to demonstrate the effect of remote faith healing

14. Multiplicity of Error • What was learned until after the fact: • Study was designed to look for differences in mortality rates • Triple-drug Anti-Retroviral therapy became common practice one month into the six month study • Only one person died – insufficient data to make conclusions • Adjusted the scope to look at HIV physical symptoms and Quality of Life • Also inconclusive • Adjusted scope again to look at Mood State scores • Control group performed better than treatment group • Adjusted scope again to look at Hospital Stays and Doctor Visits • Treatment group performed statistically significantly better • Adjusted scope to include at 23 illnesses associated with AIDS • Data hadn't been collected as part of the study design, so blindness was broken and data was aggregated from hospital charts • Sharpshooter fallacy • Spray bullets down range, then draw your target around a cluster

15. Multiplicity of Error • Keeping ourselves honest • Bonferroni correction • Don't use α, use where m is the number of comparisons • Conservative • Holm-Bonferroni correction • Compare the smallest p-value to • If it is significant, continue testing • Otherwise, stop • Compare the next smallest p-value to • Continue comparing the p-values until one is not significant, then stop • Less conservative • Others that are implemented in statistics packages (Minitab, JMP, etc…) • Tukey's HSD • Dunnet's • And many more (you can't get a Ph.D. in statistics without inventing a correction to name after yourself)

16. Biggest Lie Ever Told • Vitamin C prevents or helps cure the Common Cold • Linus Pauling (an incredibly bright, talented scientist) championed the need for megadoses of Vitamin C to prevent or cure the Common Cold (and cancer) with the publishing of his book Vitamin C and the Common Cold in 1970 • Had a whole institute setup to further study the benefits of megadosing Vitamin C • At least 30 experiments have been performed to test the ability of Vitamin C to prevent or cure the Common Cold in large groups of people (dates of experiments go from 1967-2001) • Not a single one has found a link between Vitamin C and the Common Cold