Section 6.1 ~ The Role of Probability in Statistics: Statistical Significance

1. Section 6.1 ~ The Role of Probability in Statistics: Statistical Significance Introduction to Probability and Statistics Ms. Young

2. Objective Sec. 6.1 Understand the concept of statistical significance and the essential role that probability plays in defining it.

3. Sec. 6.1 Statistical Significance • A set of measurements or observations are considered to be statistically significant if they probably DID NOT occur by chance • Ex. ~ Tossing a coin 100 times and getting 80 heads and 20 tails would be statistically significant because it probably did not occur by chance • Example 1: • Determine whether each scenario is statistically significant or not • A detective in Detroit finds that 25 of the 62 guns used in crimes during the past week were sold by the same gun shop. • This finding is statistically significant. Because there are many gun shops in the Detroit area, having 25 out of 62 guns come from the same shop seems unlikely to have occurred by chance.

4. Sec. 6.1 Example 1 Cont’d… • In terms of the global average temperature, five of the years between 1990 and 1999 were the five hottest years in the 20th century. • Having the five hottest years in 1990–1999 is statistically significant • By chance alone, any particular year in a century would have a 5 in 100, or 1 in 20, chance of being one of the five hottest years. Having five of those years come in the same decade is very unlikely to have occurred by chance alone • This statistical significance suggests that the world may be warming up • The team with the worst win-loss record in basketball wins one game against the defending league champions. • This one win is not statistically significant because although we expect a team with a poor win-loss record to lose most of its games, we also expect it to win occasionally, even against the defending league champions

5. Sec. 6.1 Example 2 • A researcher conducts a double-blind experiment that tests whether a new herbal formula is effective in preventing colds. During a three-month period, the 100 randomly selected people in a treatment group take the herbal formula while the 100 randomly selected people in a control group take a placebo. The results show that 30 people in the treatment group get colds, compared to 32 people in the control group. Can we conclude that the new herbal formula is effective in preventing colds? • Whether a person gets a cold during any three-month period depends on many unpredictable factors. Therefore, we should not expect the number of people with colds in any two groups of 100 people to be exactly the same. • In this case, the difference between 30 people getting colds in the treatment group and 32 people getting colds in the control group seems small enough to be explainable by chance. • So the difference is not statistically significant, and we should not conclude that the treatment is effective.

6. Sec. 6.1 Quantifying Statistical Significance • Determining if something is statistically significant can be obvious in some cases (i.e, 80 heads vs. 20 tails), but how do you decide if something is statistically significant if the numbers are closer (i.e., 55 heads vs. 45 tails)? • Probability is used to quantify statistical significance by determining the likelihood that a result may have occurred by chance • .05 level of significance: if the probability that something DID occur by chance is less than or equal to .05, or 5%, then it is statistically significant at the .05 level • In other words, if the probability that something did occur by chance is small (5%), then the probability that it did not occur by chance is big (95%), which means it is statistically significant because it probably did not occur by chance • .01 level of significance: if the probability that something DID occur by chance is less than or equal to .01, or 1%, then it is statistically significant at the .01 level • In other words, if the probability that something did occur by chance is small (1%), then the probability that it did not occur by chance is big (99%), which means it is statistically significant because it probably did not occur by chance • Something that is significant at the .01 level is also significant at the .05 level (since 1% is less than 5%), but something significant at the .05 level is not necessarily significant at the .01 level (since something could be significant at the .05 level if it’s under 5%, but doesn’t have to be as low as 1%)

7. Sec. 6.1 Example 3 • In the test of the Salk polio vaccine (see Section 1.1), 33 of the 200,000 children in the treatment group got paralytic polio, while 115 of the 200,000 in the control group got paralytic polio. Calculations show that the probability of this difference between the groups occurring by chance is less than 0.01. Describe the implications of this result. • The results are significant at the .01 level. This means there is a 1% chance or less that the results occurred by chance, therefore the results probably did not occur by chance which means that there is good reason to believe that the treatment works.