Power Analysis for Traditional and Modern Hypothesis Tests

1. Power Analysis for Traditional and Modern Hypothesis Tests Kevin R. Murphy Pennsylvania State University

2. Power Analysis • Helps you plan better studies • Helps you make better sense of existing studies • Is not limited to traditional null hypothesis tests • Application of power analysis to minimum-effect tests will be discussed

3. Errors in Null Hypothesis Tests True State of Affairs Your Decision

4. Power Depends On • Effect Size • How large is the effect in the population? • Sample Size (N) • You are using a sample to make inferences about the population. How large is the sample? • Decision Criteria -  • How do you define “significant” and why?

5. Power Analysis and the F Distribution • The power of most statistical tests in social sciences (e.g., ANOVA, regression, t-tests, other linear model statistics) can be evaluated via the familiar F distribution • F is a ratio of observed effect to error • F= MS treatments / MS error • F = (True Effect + Error) / Error • The larger the true treatment effect, the larger F you expect to find • If the null hypothesis is correct, E(F) = 1.0

6. How Does Power Analysis Work? In the familiar F distribution below, 95% of the values are below 2.00 (distribution for df = 7,200) F=2.0 represents cutoff for rejecting H0

7. The Noncentral F Distribution If the null hypothesis is false, the Noncentral F distribution is needed. In the Noncentral F distribution below, 75% of the values are below 2.00. Therefore, power = .25

8. A Larger Effect In the Noncentral F distribution below, in which the effect is larger, 30% of the values are below 2.00. Therefore power = .70

9. Power Functions

10. Power Functions

11. How to Increase Power Increase N • Effects of adding more subjects are not identical to those of adding more observations Increase ES • Choose a different research question • Use stronger treatments or interventions • Use better measures Use a more lenient alpha • p<.05 is driven by force of habit, not necessarily by substantive concerns

12. Effects of Implementing Power Analysis • Stronger studies • Larger samples, better measures • Fewer studies • Adequate studies are harder to do than most people realize • Less emphasis, in the long term, on null hypothesis testing

13. Conducting a Power Analysis • The classic text in this field is still one of the best sources • Cohen, J. (1998). Statistical power analysis for the behavioral sciences (2nd Ed.). Erlbaum • More current (and more accessible) sources include • Lipsey, M. (1990). Design sensitivity. Sage • Murphy, K. & Myors, B. (2004). Statistical power analysis: A simple and general model for traditional and modern hypothesis tests. Erlbaum.

14. Conducting a Power Analysis • Power Analysis software • Power and Precision - Biostat • www.PowerAnalysis.com • One-Stop F Calculator • Included in Murphy & Myors (2004) • PASS - NCSS software • www.ncss.com/pass.html

15. Conducting a Power Analysis • In planning studies, you should • Assume relatively small effects • If it was reasonable to expect a large effect, you probably don’t need to do the study or the test • Aim for power of .80 or better • Power of .50 means that significance tests have become a coin flip

16. Effect Size Conventions • In behavioral and social sciences, there are widely-followed conventions for describing small, moderate, and large effects d- standardized Percentage of mean differencevariance explained Small .20 1% Moderate .50 10% Large .80 25%

17. Applications of Power Analysis • Study planning - Given ES and , solve for N • If you wanted to compare the effects of four types of training programs and: • You expected small to moderate effects (programs account for 5% of variation in performance) • You use an  level of .05 • You need N=214 to achieve Power=.80

18. Applications of Power Analysis • Study evaluation - Given N and , solve for ES • If you wanted to compare the effects of four safety interventions and: • You have 44 subjects available • You use an  level of .05 • You will achieve Power=.80 only if the effects of interventions are truly large (accounting for 25% of the variance in outcomes)

19. Applications of Power Analysis • Making a rational choice regarding Given N and ES, solve for  • If you wanted to compare the effects of two leadership development programs and: • You have 200 subjects available • You expect a small difference (d=.20, or 1% of the variance explained by programs) • You will achieve Power=.64 using  • You will achieve Power=.37 using 

20. Moving Beyond Traditional Significance Testing • Traditional null hypotheses tests are the focus of most power analyses • These tests are deeply flawed, and there is relatively little research on the power of alternatives • Minimum effect tests represent one useful alternative

21. Nil Hypothesis Testing • Testing the hypothesis that treatments, interventions, etc. have no effect (Nil Hypothesis Test - NHT) is most common and least useful thing social and behavioral scientists do • Two problems loom largest: • Confusion over Type 1 errors • Likelihood of rejecting the null hypothesis eventually reaches 1.0, regardless of the research question

22. Type I Errors are Very Rare • Type I error - reject H0 when it is true • If H0 is never true, it is impossible to make a Type I error • If H0 is very unlikely, a Type I error is even less likely • H0 - treatment had NO effect at all • H1 - SOMETHING happened • Most things we do to minimize Type I errors lead to more Type II errors

23. This Implies • Large literature on protecting yourself from Type I errors is not really useful • NHTs yield one of two outcomes • confirm the obvious • reject H0, which you already know is likely to be wrong • confuse you • “accept” H0 even though you know it is likely to be wrong

24. In NHT, All You Need in N • As N increases, the likelihood of rejecting the nil hypothesis approaches 1.0 • Power to reject H0 does not depend all that much on the phenomenon • if N is big enough you will reject H0 • if N it is small enough, you won’t • Significance tests are an indirect index of how many subjects showed up

25. There Must be a Better Way • Stop doing significance tests (e.g., Schmidt, 1992) • Confidence intervals (e.g., APA Task Force, American Psychologist, August, 1999) • Bayesian methods (e.g., Rounet, Psychological Bulletin, 1996)

26. There Must be a Better Way • Minimum-Effect Tests • Test the hypothesis that something nontrivial happened • Murphy, K. & Myors, B. (2003) Statistical power analysis: A simple and general model for traditional and modern hypothesis tests: 2nd Ed. Mahwah, NJ: Erlbaum. • Murphy, K. & Myors, B. (1999). Testing the hypothesis that treatments have negligible effects: Minimum-effect tests in the general linear model. Journal of Applied Psychology, 84, 234-248.

27. Minimum-Effect Tests • H0 - treatments have a negligible effect (e.g., they account for 1% or less of the variance) • H1 - the effect of treatments is big enough to care about • This approach addresses the two biggest flaws of traditional tests • H0 really is plausible. Treatments rarely have zero effect but they often have negligible effects • Increasing N does not automatically increase likelihood of rejecting H0

28. Minimum-Effect Tests • With Minimum Effect Tests (METs) • Type I errors are once again possible, but can be miminized • the question asked in MET is no longer trivial • you can actually learn something by doing the test • Power Analysis work exactly the same way in MET as in NHT

29. Performing Minimum-Effect Tests • Put your test statistics in a simple, common form • e.g. F • Decide what you mean by a negligible effect • Find or create an F table based on that definition of a negligible effect - Noncentral F distribution • Proceed as you would for any traditional NHT

30. Working with the Noncentral F • Calculating or deriving noncentral F distributions was once a daunting task • Many simple calculators now available • http://calculators.stat.ucla.edu/cdf/ncf/ncfcalc.php • Noncentrality parameter ( ) •  in a measure of effect size •  = [dfh * (MSh - MSe )] / MSe

31. What Constitutes a “Negligible” Effect ? • Standards for “negligible” effects depend on the research area and on the consequences of decisions • Aspirin use accounts for very little variance in heart attacks, but the use of aspirin saves thousands of lives at minimal cost • In personnel selection, it is relatively easy to account for a large proportion of the variance in performance with simple cognitive tests, so the increase in effectiveness that is defined as negligible might be larger

32. Defining a “Negligible” Effect • Effect Size conventions are useful, but by themselves may not be sufficient • Consequences of errors must also be considered d- standardized Percentage of mean differencevariance explained Small .20 1% Moderate .50 10%

33. Power Analysis for MET:Small Effect - d=.20, PV=.01

34. Power Analysis for MET:Small Effect - d=.20, PV=.01

35. Power Analysis for MET:Small Effect - d=.20, PV=.01

36. Power Analysis for MET:Small Effect - d=.20, PV=.01

37. Errors in MET • The potential downsides of MET are: • Type I errors could actually occur • Lower power than corresponding NHT • You can reduce Type I errors by using larger samples • The loss of power is more than balanced by the fact that the hypothesis being tested is not a trivial one

38. Type I Error Rates of Minimum-Effect Tests

39. Type I vs Type II Errors • The tradeoff between Type I and Type II errors is more complicated in METs than in Nil tests • In MET, alpha is precise only if the true effect size is exactly the same as your definition of “negligible” • Type II errors more of a problem with METs • METs are less powerful than NHTs (it is easier to reject the hypothesis that nothing happened than the hypothesis that nothing important happened), but this is not necessarily a bad thing • METs place even greater premium on large samples, but small samples cause problems even where there is substantial power

40. Examples - comparing two treatments • N needed True effect PV=.05 PV=.10 Nil 149 79 MET 375 117 (1%=negligible)