Exploring Quasi-Experiments Lab 5: May 9, 2008

1. Exploring Quasi-ExperimentsLab 5: May 9, 2008 Guthrie, J.T., Wigfield, A., & VonSecker, C. (2000). Effects of integrated instruction on motivation and strategy use in reading. Journal of Educational Psychology, 92, 331-341.

2. Purpose of Research • Causal relationship • To identify whether the Concept-Oriented Reading instruction (CORI) intervention produces greater intrinsic motivation and strategy-use than traditional instruction

3. Variables of interest • The Independent Variables (causes) • Instructional intervention: • CORI intervention • traditional comparison group • The Dependent Variables (outcomes) • Intrinsic motivation (operationalized: curiosity, involvement, and preference for challenge) • Extrinsic motivation (operationalized: recognition and competition) • Strategy use (operationalized: self-report of cognitive strategies) • The Covariates • Past achievement (operationalized: standardized reading achievement scores—CBST/MAT)

4. Participant assignment • Non-random assignment into intervention and comparison groups (i.e., into classrooms) • Classrooms assigned based on comparable “subjective matching” of teachers, students, and school settings

5. Cook and Campbell’s UTOS • Units: 3rd and 5th grade low-achieving students. • Treatment: Concept-oriented reading instruction (CORI) • Observations: intrinsic and extrinsic motivation, and strategy use (also past achievement). • Setting: three different mid-Atlantic grade schools.

6. Threats to Internal ValiditySpecific to Quasi-Experiments • History • Some event related to the outcome could occur to one group but not the other • Maturation • Groups may differ in rate of change on the outcomes prior to treatment • Instrumentation • Measurement might change from pre to post test in only one group • Statistical Regression • Variable used to determine group selection may be unreliable or unstable

7. Threats to Internal ValiditySpecific to Quasi-experiments • History: A celebrity may visit the schools to discuss the importance of reading; CORI students may be more susceptible to the message and therefore show greater gains in motivation (due to the celebrity, not the intervention). • Maturation: Children in the traditional classrooms could be losing motivation at a faster rate than those in the CORI classrooms prior to treatment. • Instrumentation: Likely not an issue given the lack of pretest design. • Statistical Regression: The choosing of similar classrooms based on teacher, student, and school make-up could have been based on inaccurate, unreliable, or temporarily skewed subjective judgments.

8. AdditionalThreats to Internal Validity • Lack of pretest makes it difficult to say whether differences are due to CORI or whether the differences existed at the onset of the research • Attrition: 11% for grade 3 and 17% for grade 5 due to moving. • Resentful Demoralization: teachers or students could have shown less motivation knowing that they were not getting the treatment (p.334/ p. 47) • Compensatory Rivalry: comparison teachers may have tried harder because of the study (mentioned on p. 334 as the John Henry effect). • Third variables: Teacher expectations could have produced results • Authors did enhance their design by including extrinsic motivation (p. 332) as a nonequivalent dependent variable (p. 74).

9. Threats to Construct Validity • Diffusion of treatment: Traditional teachers frequently visited the CORI classrooms and adopted some texts used in the CORI condition. • The authors did collect video, interview, and questionnaire data regarding the use of the treatment integrity in the CORI classrooms. • Low internal consistency: Several internal consistencies were low; because reliability puts a lid on validity, this is cause for concern. • Mono-operation bias: Motivation is captured using only aspects of the Motivation for Reading Questionnaire (MRQ) when other methods may be possible. (e.g., Teacher-reports on individual students). • The authors did address more than one aspect of motivation (intrinsic and extrinsic)

10. Threats to External Validity • Schools are chosen based on need, therefore results may not generalize to less needy students/schools. • Results may not generalize to other grades • Results may not generalize to interventions that are less intensive. • Results may not generalize to areas outside of the mid-Atlantic. • Results may not generalize to participants from dissimilar schools.

11. Threats to Statistical Conclusion Validity • Violated statistical assumptions: Group administration of treatment violates assumption of independent observations (p. 49). • Addressed by analyzing the data using HLM

12. Improving the research • Proximal Similarity: The traditional instruction comparison group could have maintained integrity and differentiation from CORI instruction group. The measures could have more closely approximated intrinsic motivation. • Researchers did a good job employing similar settings and units to whom they wished to generalize. • Researchers did a good job ensuring that CORI instruction was being implemented in the CORI classrooms • Heterogeneous Irrelevancies: Triangulating measures of motivation could have improved construct validity. • The authors did use two different Units and three different Settings. • Causal explanation: Rule out third variables and include pretest.