Chapter 18 Experimental and Quasi-Experimental Research

Chapter 18 Experimental and Quasi-Experimental Research Research Methods in Physical Activity

Experimental research Experimental research attempts to establish cause-and-effect relationships. Three criteria must be present to establish cause and effect: 1. The cause must precede the effect in time. For example, the starting gun in a race precedes the runners’ beginning the race; the runners’ beginning does not cause the starting gun to go off. 2. The cause and effect must be correlated with each other. Just because two variables are correlated does not mean one causes the other; cause and effect, however, cannot exist unless two variables are correlated. 3. The correlation between cause and effect cannot be explained by another variable. Recall that the relationship between the academic performance of elementary children and shoe size was explained by a third variable, age. Research Methods in Physical Activity

Experimental research Cause and effect in must be defined terms of necessary and sufficient conditions. For example, if the condition is necessary and sufficient to produce the effect, then it is the cause. Alternative situations exist as well: ♦Necessary but not sufficient: Some related condition likely produces the effect. ♦Sufficient but not necessary: Some alternative condition is likely the cause. ♦Neither necessary nor sufficient: Some contributing condition is likely the cause. Research Methods in Physical Activity

Experimental Research • Also, remember that statistics do not establish cause and effect. • Statistical techniques can only reject the null hypothesis (establish that groups are significantly different) and identify the percentage of variance in the dependent variable accounted for by the independent variable or the effect size. • Neither of these procedures establishes cause and effect (they are necessary but not sufficient). • Cause and effect can be established only by the application of logical thinking to well-designed experiments. • This logical process establishes that no other reasonable explanation exists (reject Null for alternate hypothesis) for the changes in the dependent variable except the manipulation of the independent variable. Research Methods in Physical Activity

Experimental Research The logical process of establishing changes in the dependent variable require: ♦ The selection of a good theoretical framework ♦ The use of appropriate participants ♦ The application of an appropriate experimental design ♦ The proper selection and control of the independent variable (treatment) ♦ The appropriate selection and measurement of the dependent variable ♦ The use of the correct statistical model and analysis ♦ The correct interpretation of the results Thus, the above are the methods required to establish valid findings allowing the researcher to establish cause and effect. Research Methods in Physical Activity

Sources of Invalidity • Both internal validity and external validity are important in experiments, but they are frequently at odds in research planning and design. • Gaining internal validity involves controlling all variables so that the researcher can eliminate all rival hypotheses as explanations for the outcomes observed. • When controlling and constraining the research setting to gain internal validity, the researcher places the generalization (external validity) of the findings in jeopardy. • Thus, the researcher must decide whether it is more important to be certain that the manipulation of the independent variable caused the observed changes in the dependent variable or more important to be able to generalize the results to other populations, settings, etc. Research Methods in Physical Activity

Threats to Internal Validity in Research Design (see examples of each pp. 332-334) ♦ History: events occurring during the experiment that are not part of the treatment ♦ Maturation: processes within the participants that operate as a result of time passing (e.g., aging, fatigue, hunger) ♦ Testing: the effects of one test on subsequent administrations of the same test ♦ Instrumentation: changes in instrument calibration, including lack of agreement within and between observers ♦ Statistical regression: the fact that groups selected based on extreme scores are not as extreme on subsequent testing ♦ Selection bias: choosing comparison groups in a nonrandom manner ♦ Experimental mortality: loss of participants from comparison groups for nonrandom reasons. Research Methods in Physical Activity

Threats to Internal Validity in Research Design (see examples of each pp. 332-334) ♦ Selection–maturation interaction: the passage of time affecting one group but not the other in nonequivalent group designs ♦ Expectancy: experimenters’ or testers’ anticipating that certain participants will perform better Research Methods in Physical Activity

Controlling Threats to Internal Validity in Research Design Many threats to internal validity are controlled by making the participants in the experimental and control groups as alike as possible. This objective is most often accomplished byrandomlyassigning participants to groups. Random Assignment Randomization allows the assumption that the groups do not differ at the beginning of the experiment. The randomization process controls for history up to the point of the experiment; that is, the researcher can assume that past events are equally distributed among groups Research Methods in Physical Activity

Controlling Threats to Internal Validity in Research Design Random Assignment 3) Randomization also controls for maturation because the passage of time is equivalent in all groups. 4) Statistical regression, selection biases, and selection–maturation interaction are controlled because they occur only when groups are not randomly formed. Research Methods in Physical Activity

Controlling Threats to Internal Validity in Research Design Random Assignment Ways other than random assignment of participants to groups are sometimes used to control threats to internal validity. These Include the matched-pair technique, and within-subjects designs. Matched-Pair Technique - The matched-pair technique matches pairs of participants who are equal on some characteristic and then randomly assigns each to a different group. Within-subjects designs - The participants are used as their own controls. This means that each participant receives both the experimental and the control treatment. In this type of design, the order of treatments should be counterbalanced; that is, half the participants should receive the experimental treatment first and then the control, and the other half should receive the control treatment first and then the experimental treatment. Research Methods in Physical Activity

Controlling Threats to Internal Validity in Research Design Placebos, Blind Setups, and Double-Blind Setups Other ways of controlling threats to internal validity include placebosand blind and double-blind setups. Placebo - Method of controlling a threat to internal validity in which a control group receives a false treatment while the experimental group receives the real treatment. Blind setup - Method of controlling a threat to internal validity in which the participant does not know whether he or she is receiving the experimental or control treatment. Double-blind setup - Method of controlling a threat to internal validity in which neither the participant nor the experimenter knows which treatment the participant is receiving. Research Methods in Physical Activity

UncontrolledThreats to Internal Validity in Research Design Three Threats to internal validity remained uncontrolled by the randomization process : 1) Reactive or Interactive Effects; 2) Instrumentation; and, 3)Experimental Mortality. Reactive or interactive effects of testing can be controlled only by eliminating the pretest. Instrumentation problems cannot be controlled or evaluated by any design. Only the experimenter can control this threat to internal validity. Experimental mortality is not controlled by any type of experimental design. The experimenter can control this threat only by ensuring that participants are not lost (at all, if possible) from groups. Research Methods in Physical Activity

Threats to External Validity in Research Design (see examples of each pp. 334-336) Threats to external validity, or the ability to generalize results to other participants, settings, measures, etc: ♦Reactive or interactive effects of testing : The pretest may make the participant more aware of or sensitive to the upcoming treatment. As a result, the treatment is not as effective without the pretest. ♦ Interaction of selection bias and the experimental treatment: When a group is selected on some characteristic, the treatment may work only on groups possessing that characteristic. ♦ Reactive effects of experimental arrangements: Treatments that are effective in constrained situations (e.g., laboratories) may not be effective in less constrained settings (more like the real world). ♦ Multiple-treatment interference: When participants receive more than one treatment, the effects of previous treatments may influence subsequent ones Research Methods in Physical Activity

Controlling Threats to External Validity in Research Design External validity is generally controlled by selecting the participants, treatments, experimental situation, and tests to represent some larger population. The results of a particular treatment can be generalized to a larger group if the sample is representative, this generalization may apply only to the specific situation in the experiment. If the experiment is conducted under controlled laboratory conditions, then the findings may apply only under controlled laboratory conditions. Does the study have enough characteristics of real-world settings that participants respond as if they were in the real world ? (ie. Is ecological validity present ? ) Research Methods in Physical Activity

Types of Research Designs Types of research designs are divided into three categories: Pre - experimental designs, True experimental designs, and, Quasi-experimental designs. (not reviewed in this lecture) With the following notation: ♦ Each line indicates a group of participants. ♦ R signifies random assignment of participants to groups. ♦ O signifies an observation or a test. ♦ T signifies that a treatment is applied; a blank space in a line where a T appears on another line means that the group is a control. ♦ A dotted line between groups means that the groups are used intact rather than being randomly formed.. Research Methods in Physical Activity

Types of Research Designs Notations (cont) : ♦ Subscripts (ex. O1, T2)indicate either the order of observations and treatments (when they appear on the same line) or observations of different groups or different treatments (when they appear on different lines). For example, when the terms T1 and T2 appear on different lines, they refer to different treatments; when they appear on the same line, they mean that the treatment is administered more than once to the same group. Research Methods in Physical Activity

Types of Research Designs Pre - experimental Designs Pre-experimental design - One of three types of research design that control few of the sources of invalidity and that do not have random assignments of participants to groups: one-shot study, one-group pretest - posttest design, and static group comparison. One-shot study In a one-shot study design, a group of participants receives a treatment followed by a test to evaluate the treatment: T O This design fails all the tests of good research. All that can be said is that at a certain point in time this group of participants performed at a certain level. In no way can the level of performance (O) be attributed to the treatment (T). Research Methods in Physical Activity

Types of Research Designs • Pre - experimental Designs • Pretest - posttest design • The one-group pretest–posttest design, although very weak, is better than the one-shot design. At least we can observe whether any change in performance has occurred: • O1 T O2 • If O2 is better than O1, we can say that the participants improved. • Complications with this design : • 1) The improvement could be due to the treatment, but it could also be due to history. Some event other than the treatment (T) may have occurred between the pretest (O1) and the posttest (O2); Research Methods in Physical Activity

Types of Research Designs Pretest - posttest design Complications with this design (cont) : 2) Some event other than the treatment (T) may have occurred between the pretest (O1) and the posttest (O2); (ex. the participants may have exercised at home on the other days. ) 3) Maturation is a rival hypothesis. The participants may have gotten better (or worse) just as a result of the passage of time. 4) Testing is a rival hypothesis; the increase at O2 may be the result only of experience with the test at O1. 5) If the group being tested is selected for some specific reason, then any of the threats involving selection bias can occur. Research Methods in Physical Activity

Types of Research Designs Pre - experimental Designs Static Group Comparison The static group-comparison design compares two groups, one of which receives the treatment and one of which does not: T O1 O2 Complications with this design : 1) We do not know whether the groups were not equivalent when the study began, as indicated by the dotted line between the groups. This means that the groups were selected intact rather than being randomly formed. Therefore, we are unable to determine whether any differences between O1 and O2 are because of T or only because the groups differed initially. Research Methods in Physical Activity

Types of Research Designs Static Group Comparison Complications with this design (cont) : 2) This design is subject to invalidity because of selection biases and the selection–maturation interaction. Research Methods in Physical Activity

Types of Research Designs True Experimental Designs True experimental design - Any design used in experimental research in which groups are randomly formed and that controls most sources of invalidity. This controls for past (but not present) history, maturation (which should occur equally in the groups), testing, and all sources of invalidity that are based on nonequivalence of groups (statistical regression, selection biases, and selection–maturation interaction). Only the experimenter can make sure that nothing happens to one group (besides the treatment) and not the other (present history), that scores on the dependent measure do not vary as a result of instrumentation problems, and that the loss of participants is not different between the groups (experimental mortality). Research Methods in Physical Activity

Types of Research Designs True Experimental Designs True experimental designs include: Randomized–Groups Design Pretest–Posttest Randomized–Groups Design, and Solomon Four-Group Design Research Methods in Physical Activity

Types of Research Designs True Experimental Designs Randomized–Groups Design The randomized-groups design resembles static group comparison except that groups are randomly formed: R T O1 R O2 If the researcher controls the threats to internal validity that are not controlled by randomization, and has a sound theoretical basis for the study, and meets the necessary - and-sufficient rule, then this design allows the conclusion that significant differences between O1 and O2 are due to T. An independent t test is used to analyze the difference between O1 and O2. Research Methods in Physical Activity

Types of Research Designs True Experimental Designs Randomized–Groups Design (cont) This design can be extended to any number of levels of an independent variable: R T1 O1 R T2 O2 R O3 Here, three levels of the independent variable exist, where one is the control and T1 and T2 represent two levels of treatment. This design can be analyzed by simple ANOVA, which contrasts the dependent variable as measured in the three groups (O1, O2, O3). Research Methods in Physical Activity

Types of Research Designs True Experimental Designs Pretest–Posttest Randomized–Groups Design In the pretest–posttest randomized-groups design, the groups are randomly formed, but both groups are given a pretest as well as a posttest: R O1 T O2 R O3 O4 The major purpose of this type of design is to determine the amount of change produced by the treatment; that is, does the experimental group change more than the control group? Research Methods in Physical Activity

Types of Research Designs True Experimental Designs Pretest–Posttest Randomized–Groups Design (cont) This design threatens the internal validity through testing, but the threat is controlled because the comparison of O3 to O4 in the control group and the comparison of O1 to O2 in the experimental group includes the testing effect. Thus, although the testing effect cannot be evaluated in this design, it is controlled. This design can (best) be analyzed by repeated measures ANOVA. Research Methods in Physical Activity

Types of Research Designs True Experimental Designs Solomon Four-Group Design The Solomon four-group design is the only true experimental design to specifically evaluate one of the threats to external validity: reactive or interactive effects of testing. The design is depicted as follows: R O1 T O2 R O3 O4 R T O5 R O6 This approach combines the randomized-groups and the pretest–posttest randomized-groups designs. The purpose is explicitly to determine whether the pretest results in increased sensitivity of the participants to the treatment. Research Methods in Physical Activity

Types of Research Designs Solomon Four-Group Design (cont) R O1 T O2 R O3 O4 R T O5 R O6 This design allows: a replication of the treatment effect (is O2 > O4? and is O5 > O6?), an assessment of the amount of change due to the treatment (is O2 – O1 > O4 – O3?), an evaluation of the testing effect (is O4 > O6?), and, an assessment of whether the pretest interacts with the treatment (is O2 > O5?). Research Methods in Physical Activity

Types of Research Designs Solomon Four-Group Design (cont) In addition, no good way exists to analyze this design statistically. The best alternative (one that does not use all the data) is a 2 X 2 ANOVA set up as follows: No T T Pretested O4 O2 Unpretested O6 O5 Thus, independent variable 1 has two levels (pretested and not pretested), and independent variable 2 has two levels (treatment and no treatment). In the ANOVA, the F ratio for independent variable 1 establishes the effects of pretesting, the F for independent variable 2 establishes the effects of the treatment, and the F for interaction evaluates the external validity threat of interaction of the pretest with the treatment. End of Lecture Research Methods in Physical Activity

Chapter 18 Experimental and Quasi-Experimental Research