1 / 59

Quasi-Experiments – Outline

Quasi-Experiments – Outline. True Experiments Characteristics Threats to validity controlled by experiments Threats not controlled by experiments Obstacles to true experiments in the field Quasi-experiments The logic of quasi-experiments Non-equivalent control group design

adouglass
Télécharger la présentation

Quasi-Experiments – Outline

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Quasi-Experiments – Outline • True Experiments • Characteristics • Threats to validity controlled by experiments • Threats not controlled by experiments • Obstacles to true experiments in the field • Quasi-experiments • The logic of quasi-experiments • Non-equivalent control group design • Example – Langer & Rudin (1976) • Interrupted time-series design • Example – Campbell (1969) Quasi

  2. True Experiments - Characteristics • True experiments are characterized by: • A manipulation • A high degree of control • An appropriate comparison (the major goal of exerting control) • Manipulation in the presence of control gives you an appropriate comparison. Quasi

  3. Threats to validity controlled by true experiments • History • occurrence of an event other than the treatment Quasi

  4. Threats to validity controlled by true experiments • History • Maturation • participants always change as a function of time. Is change in behavior due to something else? Quasi

  5. Threats to validity controlled by true experiments • History • Maturation • Testing • improvement due to practice on a test (familiarity with procedure, or with testers expectations) Quasi

  6. Threats to validity controlled by true experiments • History • Maturation • Testing • Instrumentation • especially if humans are used to assess behavior (fatigue, practice) Quasi

  7. Threats to validity controlled by true experiments • History • Maturation • Testing • Instrumentation • Regression • when first observation is extreme, next one is likely to be closer to the mean. Quasi

  8. Threats to validity controlled by true experiments • History • Maturation • Testing • Instrumentation • Regression • Selection • if differences between groups exist from the outset of a study Quasi

  9. Threats to validity controlled by true experiments • History • Maturation • Testing • Instrumentation • Regression • Selection • Mortality • if exit from a study is not random, groups may end up very different Quasi

  10. Threats to validity controlled by true experiments • History • Maturation • Testing • Instrumentation • Regression • Selection • Mortality • Interactions of selection… • with History • with Maturation • with Instrumentation (ceiling effects) Quasi

  11. Note difference between these threats: • Maturation • One group; performance better on post-test than on pre-test • Interaction of Maturation & Selection • Two or more groups • Performance difference larger on post-test than on pre-test Quasi

  12. Threats to validity not controlled by experiments • Contamination • communication of information about the experiment between groups of subjects • Cook & Campbell (1979): • resentment • ‘compensatory rivalry’ • diffusion of treatment: control subjects use information given to others to change their own behavior. Quasi

  13. Contamination – an example • Craven, Marsh, Debus, & Jayasinghe (2001) • Journal of Educational Psychology • Teachers trained to improve students’ academic self-concept through praise • Internal control • External control Quasi

  14. Contamination – an example • Craven, Marsh, Debus, & Jayasinghe (2001) • Next slide shows T2 (post-test) academic self-concept scores as a function of T1 scores for control children only. Quasi

  15. 1.0 0.5 0.0 -0.5 -1.0 Low Medium High T1 acad self concept External control Internal control Internal high focus Internal low focus T2 acad self concept No diffusion Resentful demoralization? Overzealous cooperation? Low focus group consistently higher than external control Diffusion

  16. Threats to validity not controlled by experiments • Contamination • Threats to external validity • best way to deal with this is replication Quasi

  17. Threats to validity not controlled by experiments • Contamination • Threats to external validity • Hawthorne effects • changes in a person’s behavior due to being studied rather than the manipulation. • a special kind of reactivity. Quasi

  18. Hawthorne effects • Demand characteristics • cues communicated by researcher • subject’s under-standing of their role Quasi

  19. Hawthorne effects • Role of “research subject” • Is subject behaving the way he thinks a person in that role should behave? • (E.g., hypnotized person) Quasi

  20. Hawthorne effects • Orne (1962) • ‘good subjects’ think they are contributing to science by complying with researcher’s demands Quasi

  21. Hawthorne effects • What to do about Hawthorne effects? • Orne (1962): Use quasi-control subjects as “co-investigators” • They do your task, reflect on demand characteristics of the experiment. Quasi

  22. Obstacles to true experiments in the field • Sometimes, we cannot bring the phenomenon we want to study into the lab, so we have to work in the field. • Can we do experiments in the field? Quasi

  23. Obstacles to true experiments in the field • Can’t get permission from individuals in authority? • Your study may involve some time and effort on their part. But what’s in it for them? • In schools, parents also have to agree. Quasi

  24. Obstacles to true experiments in the field • Can’t get permission from individuals in authority? • Can’t assign subjects to groups randomly? • have to work with intact groups (e.g., classes in a school) Quasi

  25. Quasi-Experiments • Quasi-experiments resemble true experiments… • usually include a manipulation, and provide a comparison. • …but they are not true experiments. • lack high degree of control that is characteristic of true experiments. Quasi

  26. Quasi-Experiments • Quasi-Experiments are compromises • They allow the researcher some control when full control is not possible. Quasi

  27. Quasi-Experiments • Because full control is not possible, there may be several “rival hypotheses” competing as accounts of any change in behavior observed. • How do we convince others that our hypothesis is the right one? Quasi

  28. The Logic of Quasi-Experiments • Eliminate any threats you can • Show how each threat to validity on list given above is dealt with in your study. • Argue that others don’t apply. • using evidence or logic Quasi

  29. Two kinds of quasi-experiments • Non-equivalent control group • “non-equivalent” because not randomly assigned Quasi

  30. Two kinds of quasi-experiments • Non-equivalent control group • Interrupted time-series design • a series of observations over time, interrupted by some treatment Quasi

  31. Non-equivalent Control Group design • Control group is “like” the treatment group. • Chosen from same population • Pre- and post-test measures obtained for both groups, so similarity can be assessed. Quasi

  32. Non-equivalent Control Group design • Control group is not equivalent • subjects are not randomly-assigned to control & treatment groups • so best you can do is argue that comparison is appropriate. Quasi

  33. Non-equivalent Control Group design • If the groups are comparable to begin with, this design potentially eliminates threats to internal validity due to: • History • Maturation • Testing • Instrumentation • Regression Quasi

  34. Problems with the NECG design • Threats to validity due to interactionswith selection may not be eliminated using the NECG design. • Selection and maturation • Most likely when treatment group is self-selected (as in psychotherapy cases – people who sought help). Quasi

  35. Problems with the NECG design • Selection and maturation • Selection and history • Does one group experience some event that has a positive or negative effect (e.g., teacher of one class leaves)? Quasi

  36. Problems with the NECG design • Selection and maturation • Selection and history • Selection and instrumentation • Does one group show ceiling or floor effects? Quasi

  37. Problems with the NECG design • Selection and maturation • Selection and history • Selection and instrumentation • Regression to the mean • Are one group’s pretest scores more extreme than the other group’s? Quasi

  38. Posttest Pretest Control group Possible NECG study outcomes • both experimental and control groups show improve-ment from pretest to posttest • appears not to be any effect of the treatment Quasi

  39. Pretest Posttest Control group Possible NECG study outcomes • Looks like a treatment effect, but there may be a threat due to • selection and maturation, • selection and history Quasi

  40. Pretest Posttest Control group Possible NECG study outcomes • Selection and maturation could be a threat • Or interaction of selection and • history • testing • instrumentation • or mortality. Quasi

  41. Pretest Posttest Possible NECG study outcomes • Interaction of selection and regression looks like a serious threat here • Selection and maturation probably not a threat here. Quasi

  42. Pretest Posttest Possible NECG study outcomes • Crossover effect • Clearest evidence for an effect of the program of any of these graphs. • Selection and instrumentation not a problem – no ceiling or floor effects Quasi

  43. Quasi-experiment example • Langer & Rudin (1976) • Research conducted in retirement home. • Residents on one floor given more control over their daily lives • Residents of another floor given same interaction with staff, but no increased control. Quasi

  44. Langer & Rudin (1976) – Measures • Ratings • Self-report of feeling of control from residents • Staff assessments of mental & physical well-being, by ‘blind’ assessors • Objective measures • record of movie attendance • participation in “Guess how many jelly-beans” contest on each floor Quasi

  45. L & R (1976) – limits on control • L & R had no control over • who entered the home • who was assigned to either floor. • no control over staff hiring or firing / resigning. Quasi

  46. L & R (1976) – Possible Problems • Interaction of Selection and Maturation • even if groups have similar pretest scores, they may differ on things pretest didn’t measure • probably not a problem here – people on both floors had similar SES • assigned to floors randomly, not by health status. Quasi

  47. L & R (1976) – Possible Problems • Selection and history • suppose a popular (or unpopular) nurse left one of the floors during the study. That might influence well-being. • L & R did not address this issue. Quasi

  48. L & R (1976) – Possible Problems • Selection and history • Selection and instrumentation • did one group show ceiling or floor effects? • L & R say, no. Quasi

  49. L & R (1976) – Possible Problems • Selection and history • Selection and instrumentation • Regression • were one group’s pretest scores more extreme than the others? • L & R say, no. Quasi

  50. L & R (1976) – Possible Problems • Selection and history • Selection and instrumentation • Regression • Observer bias and Contamination • observers in the L & R study were not aware of the hypothesis. • L & R reported there was little communication between floors. Quasi

More Related