1. Statistical Significance KNR 164

2. What is Statistical Significance? • A statistical hypothesis test is a method of making decisions using data from a scientific study • In statistics, a result is called statistically significant if it has been predicted as unlikely to have occurred by chance alone, according to a pre-determined threshold probability, the significance level

3. What is a p-value? • p-value is the probability that we would have seen our data (or something more unexpected) just by chance if the null hypothesis (null value) is true • Small p-values mean the null hypothesis is unlikely given the data

4. What is a p-value? • Test to see if you can reject the null hypothesis • Many common statistical tests, such as chi-squared tests or Student's t-test, produce test statistics which can be interpreted using p-values • If you reject the null hypothesis, your results indicate that the observed result would be highly unlikely under the null hypothesis • that is, the observation is highly unlikely to be the result of random chance alone

5. What is a p-value? • Our data are so unlikely given the null hypothesis that I’m going to reject the null hypothesis! • Don’t want to reject our data – we want to accept our hypothesis and reject the null as truth (and be confident in doing this)

6. Alpha • Alpha () is the level of significance in hypothesis testing: • Alpha is a probability specified before the test is performed. • Alpha is the probability of rejecting the null hypothesis when it is true. • By convention, typical values of alpha specified in medical research are 0.05 and 0.01.

7. What does Alpha Level Mean? • Alpha () level indicates at what p-value the null will be rejected • Example: • = 0.05; calculated p-value = 0.008; rejectnull hypothesis • = 0.05; calculated p-value = 0.110; do not reject null hypothesis

8. The p-value • By convention, p-values of <.05 are often accepted as “statistically significant” in the medical literature; but this is an arbitrary cut-off • A cut-off of p<.05 ( = .05) means that in about 5 of 100 experiments, a result would appear significant just by chance

9. Hypothesis Testing The Steps: 1. Define your hypotheses (null, alternative) 2. Specify your null distribution 3. Do an experiment 4. Calculate the p-value of what you observed 5. Reject or fail to reject (~accept) the null hypothesis

10. How to interpret a p-value • P-value is significant or it is not • It is a all-or-none decision • Compared to the a priori p-value • How confident do you want to be given your result? p< .05 says 95% confident. • p = .001 is the same as p = .02 in terms of interpretation as (they are both less then p = .05) Gilbert PB, Berger JO, Stablein D, Becker S, Essex M, Hammer SM, Kim JH, DeGruttola VG. Statistical interpretation of the RV144 HIV vaccine efficacy trial in Thailand: a case study for statistical issues in efficacy trials. J Infect Dis 2011; 203: 969-975.