Understanding Statistical Analysis: Small Samples, Confidence Intervals, and Hypothesis Testing
This guide explores statistical methods for analyzing small sample sizes (less than 30) using t-distributions. It covers key concepts such as calculating sensitivity, specificity, likelihood ratios, and confidence intervals. We delve into hypothesis testing, discussing null and alternative hypotheses, Type I and Type II errors, significance levels, and p-values. Additionally, determinants of statistical power and critical regions for one-tailed and two-tailed tests are explained. Understanding these principles is crucial for effective statistical analysis in research.
Understanding Statistical Analysis: Small Samples, Confidence Intervals, and Hypothesis Testing
E N D
Presentation Transcript
PTP 560 • Research Methods Week 9 Thomas Ruediger, PT
Confidence Intervals with small samples • What is small sample size? • Less than 30 (one of those special numbers in stats) • Sampling distribution tends to spread out • Standard normal curve not adequate • Use the t-distributions • Theoretical sampling distributions • Flatter peak, wider at the tails • Approaches normal curve as sample size increases • Use values of t instead of z • Described by degrees of freedom (n-1 for confidence intervals)
Hypothesis testing • Are the differences • Representative of “real” effects? • Just by chance? • Null hypothesis ( HO) • Means are not different • Stated in terms of population parameter • μA=μB • Alternative hypothesis (H1) • Difference too large to be by chance • μA≠μB • May be directional or non-directional
Truth Ho is True Ho is False Reject Ho Type I error Correct α Decision Do Not Reject Ho Correct Type II error β
Type I Error • Significance Level (Alpha level , α level) • Your choice of how much risk you are willing to take of saying there is a difference when there really is no difference • Set this before the study • Conventionally is 0.05 • This is arbitrary, but almost always what we choose • Choose the level based on the Type I error concern
Type I Error • Probability Values (Evaluated after the study) • Probability of finding this big a difference by chance • p = .07 of this big a difference by chance • You are notstating the probability of the inverse • p = .93 that it is real difference is not appropriate • Compare p-value to alpha level if greater • Compare your p-value (calculated after the study) with your α level (set before the study) • If the p-value is lessthan α, reject the null • If the p-value is greaterthan α, fail to reject the null
Type II ErrorStatistical Power • Beta (β) • Probability of failing to reject a false HO(null hypotheses) • Β of 0.20 is 20% chance we will make a Type II error • Statistical Power • Complement of β (not compliment) • In this example 0.80 (1.00 – 0.20 = 0.80) • 80% probability of correctly rejecting the null • Before: a priori - power used to determine sample size • After: post hoc – power reported if HOnot rejected “If there was a difference, could we have found it?”
Determinants ofStatistical Power • Significance Criterion • As α increases, power increases (As α increases from 0.05 to 0.10, power increases) • Variance • As variance decreases, power increases • Sample size • As sample size increases, power increases • Effect size (difference b/w the group means) • As effect size increases, power increases
z- • z - score represents the distance between: • A sample score and • Sample mean • Divided by the standard deviation You will see this in osteoporosis scores (+2 for z- score is 2 SD away from a healthily woman mean) • z - ratio represents the distance between: • A sample mean and • Population mean • Divided by the standard error of the mean
Critical Region • That portion of the curve above and below z • If calculated z > critical z, reject HO • One or two tailed test? • Non directional hypothesis– two tailed • z of 2.00 encompasses 95.44 % (non-critical) • 4.66% is the critical area • z of 1.96 encompass 95%, • Critical region is 5%, 2.5% in each tail of a non-directional test • Directional hypothesis– one tailed • z of 1.645 encompasses 95% • Critical region is 5%, all in one tail of a directional test, while NON-direction will be 2.5% • Practically, you are disregarding everything in the other tail • Table A.1 back of P & W
Figure A: Intervention 1 is different than Intervention 2 Figure B: Intervention 1 is less effective than Intervention 2
Parametric Statistics • Used to estimate population parameters • Based on assumptions • Randomly drawn from a normally distributed population • Variances in the samples equal (at least roughly) • Interval or ratio scale • Classically, if assumptions violated, use non-parametric tests • Many view parametric stats as Robust enough to withstand even major violation
t-test • Examines two means • Two groups • Two conditions/two performances • Statistical significance based on • Difference in the means • Between the groups • The effect size • Variance • Within the groups • How variable are the scores Fig 19.1
t-test • Based on a ratio • Difference between group means/Variability within groups • Difference between means • Treatment effect and error variance • One mean- second mean and variability between the groups. In both the numerator and denominator of t-test ratio, so holds it to zero. • Variability within groups • Error variance alone • Equal and unequal variances affect t-ratio • SPSS and most other packages automatically test for this. Where? • Ratio can be written: Treatment effect and error variance/Error variance NOTE: Error variance • Not mistakes • Is anything that is not due to the independent variable
t-test • If the null is true • Ratio reduces to: Error/Error • The bigger the difference - The bigger the ratio How does the ratio get bigger? • This ratio is compared to the critical • Determines significance • Is the ratio significant? • Based on critical value (but now t instead of z) • Entering arguments (Table A.2) • Alpha level (almost always 0.05 • Degrees of freedom ( one or a few less than n ) • CI can be constructed for where the true mean difference lies
t-test • Independent t-test • Usually random assignment • Can be convenience assignment • No inherent relationship between the groups • Degrees of freedom = total sample size – 2 • Paired t-test • An inherent relationship between the groups • Self (repeated measure test) • Twins • Difference scores for each pair compared • Degrees of freedom = number of paired scores – 1 • Find this in P & W or in an SPSS output table – don’t calculate for this class!
ANOVA • Examines three (or more) means • Three (or more) groups • Three (or more) conditions/ three (or more) performances • Statistical significance based on • Difference in the means • Between the groups • The effect size • Variance • Within the groups • How variable are the scores • This should sound familiar
ANOVA • Based on ratio • Treatment effect and error variance/Error variance • For ANOVA it is the F-ratio • Derived from the Sum of Squares (SS) • Larger the SS the larger the ______________? • Calculate SS (each score minus sample mean, square each result, sum them) • Then determine the Mean Square (MS) • MSb = SSb/dfb(dfb= one less than the number of groups) • MSe = SSe/dfe (dfe = total N – number of groups) • F statistic is the ratio = MSb/Mse • Ratio of the between groups variance to error variance • Find this in P & W or in an SPSS output table – don’t calculate for this class!
