1 / 51

Experimental Design, Statistical Analysis

Experimental Design, Statistical Analysis. CSCI 4800/6800 University of Georgia March 7, 2002 Eileen Kraemer. Research Design. Elements: Observations/Measures Treatments/Programs Groups Assignment to Group Time. Observations/Measure. Notation: ‘O’ Examples: Body weight

Télécharger la présentation

Experimental Design, Statistical Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Experimental Design, Statistical Analysis CSCI 4800/6800 University of Georgia March 7, 2002 Eileen Kraemer

  2. Research Design • Elements: • Observations/Measures • Treatments/Programs • Groups • Assignment to Group • Time

  3. Observations/Measure • Notation: ‘O’ • Examples: • Body weight • Time to complete • Number of correct response • Multiple measures: O1, O2, …

  4. Treatments or Programs • Notation: ‘X’ • Use of medication • Use of visualization • Use of audio feedback • Etc. • Sometimes see X+, X-

  5. Groups • Each group is assigned a line in the design notation

  6. Assignment to Group • R = random • N = non-equivalent groups • C = assignment by cutoffs

  7. Time • Moves from left to right in diagram

  8. Types of experiments • True experiment – random assignment to groups • Quasi experiment – no random assignment, but has a control group or multiple measures • Non-experiment – no random assignment, no control, no multiple measures

  9. Design Notation Example Pretest-posttest treatment comparison group randomized experiment

  10. Design Notation Example Pretest-posttest Non-Equivalent Groups Quasi-experiment

  11. Design Notation Example Posttest Only Non-experiment

  12. Goals of design .. • Goal:to be able to show causality • First step: internal validity: • If x, then y AND • If not X, then not Y

  13. Two-group Designs • Two-group, posttest only, randomized experiment Compare by testing for differences between means of groups, using t-test or one-way Analysis of Variance(ANOVA) Note: 2 groups, post-only measure, two distributions each with mean and variance, statistical (non-chance) difference between groups

  14. To analyze … • What do we mean by a difference?

  15. Possible Outcomes:

  16. Measuring Differences …

  17. Three ways to estimate effect • Independent t-test • One-way Analysis of Variance (ANOVA) • Regression Analysis (most general) • equivalent

  18. Computing the t-value

  19. Computing the variance

  20. Regression Analysis Solve overdetermined system of equations for β0 and β1, while minimizing sum of e-terms

  21. Regression Analysis

  22. ANOVA • Compares differences within group to differences between groups • For 2 populations, 1 treatment, same as t-test • Statistic used is F value, same as square of t-value from t-test

  23. Other Experimental Designs • Signal enhancers • Factorial designs • Noise reducers • Covariance designs • Blocking designs

  24. Factorial Designs

  25. Factorial Design • Factor – major independent variable • Setting, time_on_task • Level – subdivision of a factor • Setting= in_class, pull-out • Time_on_task = 1 hour, 4 hours

  26. Factorial Design • Design notation as shown • 2x2 factorial design (2 levels of one factor X 2 levels of second factor)

  27. Outcomes of Factorial Design Experiments • Null case • Main effect • Interaction Effect

  28. The Null Case

  29. The Null Case

  30. Main Effect - Time

  31. Main Effect - Setting

  32. Main Effect - Both

  33. Interaction effects

  34. Interaction Effects

  35. Statistical Methods for Factorial Design • Regression Analysis • ANOVA

  36. ANOVA • Analysis of variance – tests hypotheses about differences between two or more means • Could do pairwise comparison using t-tests, but can lead to true hypothesis being rejected (Type I error) (higher probability than with ANOVA)

  37. Between-subjects design • Example: • Effect of intensity of background noise on reading comprehension • Group 1: 30 minutes reading, no background noise • Group 2: 30 minutes reading, moderate level of noise • Group 3: 30 minutes reading, loud background noise

  38. Experimental Design • One factor (noise), three levels(a=3) • Null hypothesis: 1 =2 =3

  39. Notation • If all sample sizes same, use n, and total N = a * n • Else N = n1 + n2 +n3

  40. Assumptions • Normal distributions • Homogeneity of variance • Variance is equal in each of the populations • Random, independent sampling • Still works well when assumptions not quite true(“robust” to violations)

  41. ANOVA • Compares two estimates of variance • MSE – Mean Square Error, variances within samples • MSB – Mean Square Between, variance of the sample means • If null hypothesis • is true, then MSE approx = MSB, since both are estimates of same quantity • Is false, the MSB sufficiently > MSE

  42. MSE

  43. MSB • Use sample means to calculate sampling distribution of the mean, = 1

  44. MSB • Sampling distribution of the mean * n • In example, MSB = (n)(sampling dist) = (4) (1) = 4

  45. Is it significant? • Depends on ratio of MSB to MSE • F = MSB/MSE • Probability value computed based on F value, F value has sampling distribution based on degrees of freedom numerator (a-1) and degrees of freedom denominator (N-a) • Lookup up F-value in table, find p value • For one degree of freedom, F == t^2

  46. Factorial Between-Subjects ANOVA, Two factors • Three significance tests • Main factor 1 • Main factor 2 • interaction

  47. Example Experiment • Two factors (dosage, task) • 3 levels of dosage (0, 100, 200 mg) • 2 levels of task (simple, complex) • 2x3 factorial design, 8 subjects/group

  48. Summary table SOURCE df Sum of Squares Mean Square F p Task 1 47125.3333 47125.3333 384.174 0.000 Dosage 2 42.6667 21.3333 0.174 0.841 TD 2 1418.6667 709.3333 5.783 0.006 ERROR 42 5152.0000 122.6667 TOTAL 47 53738.6667 • Sources of variation: • Task • Dosage • Interaction • Error

  49. Results • Sum of squares (as before) • Mean Squares = (sum of squares) / degrees of freedom • F ratios = mean square effect / mean square error • P value : Given F value and degrees of freedom, look up p value

  50. Results - example • Mean time to complete task was higher for complex task than for simple • Effect of dosage not significant • Interaction exists between dosage and task: increase in dosage decreases performance on complex while increasing performance on simple

More Related