Discussion & Conclusions

Basic research designs. Phenomenon • This is a PowerPoint show. • Click through it by pressing any key. • Focus & think about each point; do not just passively click. • Note key words and phrases Theory Hypothesis Methods & data Results Discussion & Conclusions Dr. David J. McKirnan, University of Illinois at Chicago, Psychology; mckirnanuic@gmail.com

Basic research designs. Phenomenon Want to print this file for taking notes? • This file is now in your downloads folderin "show" format (ppsx). • To convert it to a printing format (pptx): • Click 'esc' to leave this file • Open PowerPoint • Under “file” click “open”, browse to this file, click “open” • In the open file click “save as” and save as pptx. • To print in notes format: • Go to “file”  “print’. • In the dialogue box click “print what?”. • Click “Handouts (3 slides per page)” • Then, come back and re-run the show Theory Hypothesis Methods & data Results Discussion & Conclusions

Basic research designs. Phenomenon This module addresses: Designs without a control group Theory Variations on control group designs Hypothesis Methods & data The overall structure of an experiment Results Discussion & Conclusions

Week 3; Experimental designs Basic experimental designs • This module overviews the core elements of an experimental research design. • We will discuss “pre-experimental” designs • These typically have no control group or may use existing groups • They are often used in preliminary or exploratory research • “True” experiments have several key characteristics: • A control group • Random assignment of participants to groups • Standardized or uniform procedures for each group

Experimental designs and validity • We will discuss internal and external validity. • Internal validity • In experiments we manipulate (induce…) the Independent Variable. • We then measure the Dependent Variable. • Experimental hypothesis: the outcome (the level of the Dependent Variable) is caused by – and only by – the Independent Variable. • Internal validity: How confident are we that the outcome was due only to the Independent Variable. • Confound: A variable other than the IV that caused or influenced the result. • Did the participants in the experimental v. control groups differ on something other than the IV? • Were the procedures biased in some way…? Confound

Experimental designs and validity • We will discuss internal and external validity. • External validity • Experimental participants are a sample of the larger population. • The experimental manipulation attempts to accurately induce the Independent Variable. • The outcome measure represents the Dependent Variable. • The experiment is conducted in a specific physical or cultural setting. • External validity: • Does the research sample accurately represent the larger population? • Do the exp. manipulation and outcome measures accurately represent the concepts underlying the Independent & Dependent Variables? • Is the experimental setting representative of how these processes work in nature?

External validity: summary The Independent Variable Is the sample representative of the larger population? The research Sample: The Dependent Variable The research Setting: Does the outcome measure represent what we are trying to explain? Is this typical of the natural settings where the phenomenon occurs? The study structure & context Does the experimental manipulation actually create the phenomenon you are interested in?

Overview: Basic Designs • “Pre-experimental” designs: no control group Experimental Observe1 Treatment Observe2 Post-Test Only Design Pre- Post- Test Design Group assignment Pre-test Experimental manipulation Outcome

Basic Designs • “Pre-experimental” designs: no control group Experimental Observe1 Observe1 Treatment Observe2 Post-Test Only Design Pre- Post- Test Design True (or Quasi-)experimental designs with a control group “After only” Control group design Pre- Post- Group Comparisons Group assignment Pre-test Experimental manipulation Outcome Control Control Observe2

Basic Designs • “Pre-experimental” designs: no control group Post-Test Only Design Experimental Observe1 Treatment 1 Observe2 Pre- Post- Test Design Control Experimental Observe1 Treatment 2 Observe2 True (or Quasi-)experimental designs with a control group Control Observe1 Observe2 “After only” Control group design Pre- Post- Group Comparisons Multiple group comparison Group assignment Pre-test Experimental manipulation Outcome

“Pre-experimental” designs Post-Test Only Design Group Treatment Measure Only 1 group - typically an existing group: no selection or assignment occurs. Experimental intervention (“Treatment”) may or may not be controlled by the researcher. Use for naturally occurring or system-wide events (e.g., group trauma, government policy change, etc.). Measurement may or may not be controlled by the researcher. Pre- Post- Test Design Group Measure1 Treatment Measure1 Only one group; • only group available? • naturally occurring intervention? Measurements given to all participants at baseline & follow-up All participants get the same treatment, which may or may not be controlled by the researcher.

“Pre-experimental” Designs (2) Advantage of “Post-” & “Pre- Post-” Designs: Allow us to study naturally occurring interventions. • e.g., test scores before and after some school change, • Crime rates after a policy change, etc. • Having both Pre- and Post measures allows us to examine change.

“Pre-experimental” Designs (2) Disadvantage of “Post-” & “Pre- Post-” Designs: No control group = many threats to internal validity. Maturation: Participants may be older / wiser by the post-test History; Cultural or historical events may occur between pre- and post-test that change the participants Mortality: Participants may non-randomly drop out of the study Regression to baseline: Participants who are more extreme at baseline look less extreme over time as a statistical confound. Reactive Measurement:Scores may change simply due to being measured twice, not the experimental manipulation.

Experiments “After only” Control group design Control Experimental Treatment 2 Observe2 Adds a control group. Either… Observed Groups: • Naturally occurring (e.g., Class 1. v. Class 2) or • Self-selected (sought therapy v. did not…). Assigned Groups: • Randomlyassign participants to experimental v. control group, or • Matchparticipants to create equivalent groups. Measure Dependent Variable(s) only at follow-up. Use experimental or standard measures (e.g., grades, census data, crime reports). Control Observe2

Advantages of experimental design “After only” Control group design Control Experimental Treatment 2 Observe2 Control Observe2 Advantage: Lessens the likelihood of confounds or threats to internal validity. • Control group • Random assignment Disadvantage: Existing or self-selected groups may have confounds. No baseline or pre- measure available: • We cannot assess change over time. • We cannot assess whether the groups are equivalent at baseline.

Basic Designs: True experiments (2) Pre- Post- Group Comparisons (most common study design) Group 1 Measure 1 Group 2 Measure 1 Two groups: Observed (quasi-experiment) or Assigned (true experiment). Baseline (“pre-test”) measure of study variables and possible confounds.

Basic Designs: True experiments (2) Pre- Post- Group Comparisons (most common study design) Group 1 Measure 1 Treatment Measure 2 Control Group 2 Measure 1 Measure2 The group getting the experimental condition is contrasted with a control group.. “Post-test” follow-up of dependent variable(s); • Simple outcome • Change from baseline.

Basic Designs: True experiments (2) Pre- Post- Group Comparisons (most common study design) Group 1 Measure 1 Treatment Measure 2 Control Group 2 Measure 1 Measure2 Advantages: Pre-measure assesses baseline level of Dependent Variable • Allows researcher to assess change • Can find matched pairs of participants and assign each to different groups (rather than random assignment). • Can assess whether groups are equivalent at baseline. Disadvantage: Highly susceptible to confounds if using observed or self-selected groups.

More Complex Experimental Designs Multiple group comparison Group 1 Measure1 Treatment #1 Measure1 Group 2 Treatment #2 Control Measure1 Group 3 • 3 (or more) groups • Typically formed by Random assignment. Multiple experimental groups, e.g. • Low drug dose, • High drug dose, • Placebo. or • Male therapist, • Female therapist, • Wait list control.

More Complex Experimental Designs Multiple group comparison Group 1 Measure1 Treatment #1 Measure2 Measure1 Group 2 Treatment #2 Measure2 Control Measure1 Group 3 Measure2 Compare: • Level 1 of independent variable from Level 2 • Either / both experimental groups from control grp.

More Complex Experimental Designs Multiple group comparison Group 1 Measure1 Treatment #1 Measure2 Measure1 Group 2 Treatment #2 Measure2 Control Measure1 Group 3 Measure2 Advantages: Test dose or context effects: • Drug doses, amounts of psychotherapy, levels of anxiety, etc. Increasing dose effect can be tested against no dose. • Diverse conditions to test 2nd hypotheses or confounds, e.g., therapy delivered by same sex v. opposite sex therapist. Disadvantage: • More costly and complex. • Potential ethical problem with a “no dose” (or very high dose) condition.

Core components of a research study We will use this framework to think about the basic elements of an experiment. What instructions do we give? What experimental tasks will participants be performing? What measures might we be taking? Experimental & control groups get different conditions. We hypothesize that this manipulation “causes” the outcome. What outcomes are we measuring? What is the experiment trying to explain? We will have at least one Experimental Group and a Control Group. How do we assign participants to be in one or the other? Who is in our research study? How did we recruit or sample them?

Experimental design overview We recruit a sample of participants from the larger population. We randomly assign them to groups to ensure the groups are equivalent at baseline. Procedures for all groups should be exactly the same… …except the experimental manipulation, i.e., the Independent variable. Hypothesis: The outcome or Dependent Variable varies only by group.

Overview of true experimental designs Experimental group Control group

Overview: experimental designs Does the sample well represent the population? • Was recruitment biased? • Is the sample size large enough? What form of validity is threatened by sample bias? External validity Random selection What can we do to avoid that threat?

Overview: experimental designs Does the sample well represent the population? Are the groups equal at baseline? • Did participants Self-select (in or out) of the study? • Did we use existing groups? External validity Random selection Internal validity Random Assignment Validity Threat? Solution?

Overview: experimental designs • Do both groups have the same expectations? • Are participants (and researchers) really blind? • Do we treat both groups the same? Does the sample well represent the population? Are the groups equal at baseline? Procedures the same for all groups? Validity Threat? Solution? External validity Random selection Internal validity Random Assignment Internal validity: Lack of confounds

Overview: experimental designs • Does the operational definition really express the construct we are interested in? • Have we given the correct dose of the IV? Procedures the same for all groups? Does the sample well represent the population? Are the groups equal at baseline? Independent variable faithfully reflects the construct? External validity Random selection Internal validity Random Assignment Validity Threat? Solution? Internal validity: Lack of confounds External Validity Correct IV?

Overview: experimental designs Procedures the same for all groups? Does the sample well represent the population? Are the groups equal at baseline? Independent variable faithfully reflects the construct? Groups really different at outcome? • Is any difference we see actually statistically significant (reliable & meaningful)? • …or it is due to chance alone.. External validity Random selection Internal validity Random Assignment Internal validity: Lack of confounds External Validity Correct IV? Internal Validity: Statistical testing Validity Threat? Solution?

Overview: experimental designs Does the sample well represent the population? Are the groups equal at baseline? Procedures the same for all groups? Independent variable faithfully reflects the construct? Groups really different at outcome? External validity Random selection Internal validity Random Assignment Internal validity: Lack of confounds External Validity Correct IV? Internal Validity: Statistical testing

Overview: key terms • Experimental design key elements • Control group v. non-controlled designs • Threats to internal validity: • Maturation • History • Mortality • Regression to baseline • ReactiveMeasurement • “Pre-experimental” designs • Pre-post designs • Multiple group comparisons

Overview: experimental designs Does the sample well represent the population? Are the groups equal at baseline? Procedures the same for all groups? Independent variable faithfully reflects the construct? Groups really different at outcome? External validity Internal validity Internal validity External validity Internal validity

Discussion & Conclusions