Design (2): Experimental designs

1. Design (2): Experimental designs Learning outcomes • State the characteristics of an experiment • State and define different types of experimental design

2. Outline • What is an experiment? • Elements of experimental design • Setting • Experimental manipulation • Some elementary designs

3. Experiment • Defining characteristics: • Establish cause-effect relationship between IV(s) and DV as stated in experimental hypothesis • Manipulation of IV(s) • Random assignment of participants to levels (categories) of the IV(s) • If no random assignment then quasi-experiment • If neither manipulation nor random assignment then non-experiment

4. Setting • Laboratory experiment • Setting especially created for the study • Potentially high internal validity, but at the expense of external validity • Experimental realism: an experimental setting in which participants become involved, e.g. computer games and usability tests (http://www.useit.com/alertbox/20050214.html) • Field experiment – existing setting • Potentially, but not necessarily, high external validity • Mundane realism: research setting that is likely to occur in the normal course of participants’ lives, i.e. in the ‘real world’; however, this is NOT a guarantee for importance of events to or involvement of participants • ‘[The] purpose [of an experiment] is to remake the outside world so that it duplicates as closely as possible the experimental world’ (Henshel)

5. Manipulation checks • Check whether the manipulation of the IV had its intended effects • Example: • IV frustration; DV aggression • Check that the manipulation of frustration is effective, that an allegedly frustration-inducing treatment - e.g. a frustrating task - increases the level of frustration • Possible unintended effect: the measurement of the effectiveness of the IV (the check) may affect the DV indirectly through other variables, including manipulation check • Therefore, the effectiveness of the manipulation of should be checked separately in advance, in a pilot study, in order to avoid effects of the check on the DV

6. Independent measures versus repeated measures • Independent measures/between groups/independent groups/unrelated • Separate groups for each of the different experimental conditions for one ore more IVs • Each participant tested once • Repeated measures/within subjects/related • Each participant exposed to all the conditions for one or more IVs • Each participant tested several times • Mixed designs • A combination of independent-measures IVs and repeated-measures IVs

7. Independent measures • Make sure there as few differences between the groups as possible, e.g. by randomization or matching • Advantages • Simplicity – allocate each participant to one condition • Less chance of practice and fatigue effects • Useful if it is impossible for each participant to take part in all experimental conditions • Disadvantages • Expense in terms of time, effort and participant numbers • Insensitivity to experimental manipulations as a result of individual differences • Some examples of independent-measures designs follow

8. 1 Treatment-control. Post-measure only • Control group: unethical to have (withholding treatment) or unethical NOT to have (no comparison)? • Validity: • Major threats to internal validity controlled by having a control group • Because no pre-measure is taken, no sensitisation by pre-testing and no interaction of pre-testing with treatments • Other aspects of validity depend on specific details of the design • Analysis: unrelated t test (interval DV), Mann-Whitney U test (ordinal DV) or simple regression (nominal IV, interval DV)

9. 2 Categorical or continuous IV. Post-measure only • Design 1 is a special case of Design 2 • Analysis: one-way independent measures analysis of variance (interval DV), Kruskal-Wallis test (ordinal DV) or multiple regression (nominal IV, interval DV)

10. 3 Treatment-control. Pre-measures and post-measures • Validity: • Internal validity as for Design 1 • Because pre-measure is taken, possible sensitisation by pre-testing and possible interaction of pre-testing with treatments • Analysis: analysis of co-variance; analysis of change scores is NOT appropriate

11. 4 Treatment and concomitant variables • Concomitant variable (CV): related to the DV; participants’ attributes relevant to outcome measure • Purpose: control for CV (‘individual differences’ variable) and thereby reduce error term in analysis • Design 3 is a special case of Design 4 • The CV should not be influenced by treatment and should be measured before treatment is administered • Analysis: analysis of co-variance

12. 5 Factorial designs • At least one IV needs to be manipulated and participants need to be randomly assigned to treatments • Aim for equal numbers of participants in the non-manipulated IV (if there is one) when assigning levels of the IV to treatments, e.g. gender (‘stratified randomization’) • Analysis: factorial analysis of variance, e.g. two-way independent measures analysis of variance (ANOVA) • ‘Main’ effect of IV1 (ignoring the effect of IV2) • ‘Main’ effect of IV2 (ignoring the effect of IV1) • ‘Interaction’ effect (is the effect of IV1 the same across all levels of IV2?)

13. 6 Solomon four-group design • Two IVs: pre-measure (pre-test or not) and treatment (treatment given or not) • Can establish whether pre-measure interacts with treatment • More than one analysis required

14. 7 Attribute-treatment interaction • Attribute: participants’ characteristic that may moderate the effect of treatment • Equivalent to Design 4, but with different purpose • Purpose: establish if an individual difference IV (attribute, A) interacts with treatment • Analysis: analysis of co-variance

15. Repeated measures • Each participant acts as their own control • Advantages • Economy • Sensitivity • Disadvantages • Order effects and carry-over effects from one condition to another; solution: counterbalancing, e.g. using Latin squares • The need for conditions to be reversible • Some examples of repeated-measures designs follow

16. Some repeated measures designs • Repeated measures, post-test only, two conditions; analysis: related t test (interval DV) or Wilcoxon test (ordinal DV) • Repeated measures, post-test only, more than two conditions; analysis: one-way RM ANOVA (interval DV) or Friedman test (ordinal DV) • Factorial repeated measures; analysis: e.g. two-way RM ANOVA • Two-factor mixed design; this is a design in its own right, but is sometimes called repeated measures, e.g. in SPSS; analysis: two-way mixed ANOVA

17. Preparation for next practical class • Study key experimental research designs • Reading: Pedhazur: Ch. 12 • Clark-Carter: Ch. 1 • Lecture notes

18. Summary • Experimental designs differ from quasi-experimental and non-experimental designs • Experiments can be conducted in different types of setting • Manipulation checks are important and should be conducted in a pilot study • Various elementary designs differ in their validity and data analysis