Review bootstrap and permutation

1. Review bootstrap and permutation

2. Main points • Definition of confidence intervals • Definition of p-value • Why use bootstrap, why use permutation tests? • What are the differences between bootstrap and permutation tests? • How to use bootstrap? • How to use permutation tests?

3. Definition of confidence intervals • A confidence interval represents the precision of the estimation of a test statistic • If the same experiment was replicated a hundred times, the 95% CI would, on average, contain the estimated TS in 95 of these samples.

4. Definition of p-value • A p-value represents the probability of observing a TS this extreme or more extreme if the null hypothesis is true

5. Why use bootstrap, why use permutation tests? If one of those is unusual or unknown, bootstrap or permutation is useful 5

6. Test statistics • Is not implemented in standard software • Is estimated by a single value in the whole sample • Is a relationship between two TS

7. Null hypothesis • Is different from zero in a way that is hard to quantify • Example: If you are expecting a certain variable to explain more than 50 percent of a different variable, such that the r squared is greater than 50. We would want to bootstrap the r-statistics and see whether the confidence interval includes 50, or values below 50. If the confidence interval does not, then we can say that our hypothesis is supported. If it does, then we cannot reject the null.

8. Underlying distribution • Is unknown because TS was unknown • Is unknown because conditions of applications for parametric tests do not seem to be met • Is known for the null hypothesis but seems likely to be different for the alternative hypothesis p-value should be correct but CI will be incorrect.

9. Differences between bootstrap and permutation tests • Bootstrap • estimates confidence interval, bias and standard error • Simulates data under the alternative hypothesis • Sampling is done with replacement of subjects • Many bootstrap samples because of replacement • Permutation tests • estimates p-value and distribution under the null. • Simulates data under the null hypothesis • Sampling is done without replacement of subjects • Finite number of potential permutation samples

10. Main points: how to • How to bootstrap a test statistics • Determine the test statistic of interest (must be a single value) • What is randomly sampled? How many subjects in the bootstrap samples (with or without replacement)? How many bootstrap samples • Examine histogram of TS* with TS (the observed TS), average of TS*, and boundaries of percentile and bca CI • Interpret results

11. Main points: how to • How to use a permutation test • Determine the test statistic of interest (must be a single value) • What must be shuffled in order to simulate what happens under the null hypothesis? How many subjects in the permutation samples (with or without replacement)? How many permutation samples? • Examine histogram of TS* with TS (the observed TS) • Compute 2 p-values if possible: one-tailed, two-tailed • Interpret results

12. Extra assignment • Read the article by Zentner et al. (2007). • If an hypothesis is tested by either bootstrap or permutation test, describe in details: • the hypothesis, • How the hypothesis was operationalized • The procedure, • The results • The interpretation of the results.