Education 795 Class Notes

1. Education 795 Class Notes Data Analyst Pitfalls Difference Scores Effects Sizes Note set 12

2. Today’s Agenda • Announcements (ours and yours) • Data Analyst Pitfalls • Difference Scores • Effect Sizes • Multiple Comparisons

3. Data Analyst • We meet the qualitative paradigm and position ourselves in the research, quantitative discourse now includes how we the researcher affects the following: • What is to be investigated • How it is to be done • What are the facts? Hypotheses? • What are the findings? Intepretations?

4. Pitfalls • Improper analyses used • We leave out a lot of the details that enable others to properly evaluate the research • Use rules that have no consensus in the field • Some try to replicate with the same data to refute results… (we’ve seen this before) • We often ignore Validity and it is actually the crux of what we conclude • We given limited attention to Reliability

5. Difference Scores • The most common difference score • Calculate Pretest-Postest for each subject • Attain mean difference scores for treatment and comparison groups • Test the difference of the difference scores for statistical signficance • Limitation • This requires the pre and post test to have the same factor structure.

6. Problems with Difference Scores • Often referred to as ‘gain scores’ • Improvement ranges are often not equal for individuals along the continuum. • Example: Room for improvement is greater for those that start lower on the pre-test scale. • If there is a ceiling effect (too easy) or a floor effect (too hard) difference scores are meaningless because there will likely be no change • Short time intervals often don’t allow change

7. Alternatives to Raw Gain Scores • Standardized gain scores • Residualized gain scores • ANCOVA design predicting the post-test controlling for the pre-test • Here we lose some valuable information about individual change

8. Effect Sizes • Effect sizes are used to refer to magnitude, importance and meaningfulness. • Cohen (1988) defined ES as “the degree to which the phenomenon is present in the population”. • For Cohen’s d (specifically designed to asses the difference between groups), the rule is .2 small, .5 medium and .8 as large effects. • For correlation coefficients, .1 small, .3 medium, .5 large effects

9. Connecting ES to Power • Cohen (1962) showed that the median power to detect small, medium and large effects was .17, .46 and .89. • In other words, for a small effect, holding sample sizes constant, a test will only correctly reject the null hypothesis 17% of the times • For large effects present in the population, tests will correctly reject the null hypothesis 89% of the time. • Note that large effects are rarely found in sociobehavioral research

10. Effect Sizes • There are two major classes of effect sizes (not counting a third "miscellaneous" category described by Kirk (1996)): • (a) variance-accounted-for effect sizes analogous to a squared correlation coefficient • (b) standardized mean differences

11. Types of Effect Sizes • Squared Multiple Correlation Coefficient • We’ve seen this before, it is R2 • R2=Sum of Squares Regression (Effect) /Sum of Squares Total • Percent of variance in the outcome explained • Note there is no effect of sample size on this statistic. • This statistic is similar to h2 or Eta squared for ANOVA. • Note we can calculate h2 for each separate main effect in an ANOVA table.

12. Effect Sizes for Means • 1. Cohen's d = M1 - M2 / spooled     where spooled = Ö[((s1)²+ (s2)²) / 2] • Example: Assume equal n’s. M1 = 50, M2 = 60, s1=10, s2=15 • spooled = sqrt((102 + 152)/2) = sqrt(325/2) = sqrt(162.5) = 12.7 • Cohen’s d=60-50/12.7 = .78 ---LARGE EFFECT

13. Effect Size for Means • We can also calculate an effect size by the t-statistic. • Cohen's d = 2t / Ö(df) • In 1999, the APA Task Force on Statistical Inference published its report:http://www.apa.org/journals/amp/amp548594.html

14. Laptop Project • In groups of 4 • Run a Regression using at least one group membership variable. Use the descriptive statistics to calculate an effect size for the group, compare that to the statistical significance. • Present your results to the class.