Experiment Basics: Control

1. Experiment Basics: Control Psych 231: Research Methods in Psychology

2. Due this week in labs - Group project: • Methods sections • IRB worksheet (including a consent form) • Recommended/required: • Questionnaires/examples of stimuli, etc. – things that you want to have ready for pilot week (week 10) • Group Project ratings sheet • Exam 2 two weeks from today Announcements

3. Mythbusters examine: Yawning • What sort of sampling method? • Why the control group? • Should they have confirmed? • Probably not, if you do the stats, with this sample size the 4% difference isn’t big enough to reject the null hypothesis Experimental Control

4. Our goal: • To test the possibility of a systematic relationship between the variability in our IV and how that affects the variability of our DV. • Control is used to: • Minimize excessive variability • To reduce the potential of confounds (systematic variability not part of the research design) Experimental Control

5. T = NRexp + NRother + R Nonrandom (NR) Variability • Our goal: • To test the possibility of a systematic relationship between the variability in our IV and how that affects the variability of our DV. the variability in our IV NRexp: Manipulated independent variables (IV) • Our hypothesis: the IV will result in changes in the DV NRother: extraneous variables (EV) which covary with IV • Condfounds Random (R) Variability • Imprecision in measurement (DV) • Randomly varying extraneous variables (EV) Experimental Control

6. Absence of the treatment Control group (NRexp = 0) Treatment group NR NR other other NR “perfect experiment” - no confounds (NRother = 0) R R exp • Variability in a simple experiment: T = NRexp + NRother + R Experimental Control: Weight analogy

7. Difference Detector NR R R exp Our experiment is a “difference detector” • Variability in a simple experiment: T = NRexp + NRother + R Control group Treatment group Experimental Control: Weight analogy

8. NR R R exp • If there is an effect of the treatment then NRexp will ≠ 0 Control group Treatment group Difference Detector Our experiment can detect the effect of the treatment Experimental Control: Weight analogy

9. Potential Problems • Confounding • Excessive random variability Difference Detector Things making detection difficult

10. Co-vary together • Confound • If an EV co-varies with IV, then NRother component of data will be present, and may lead to misattribution of effect to IV IV DV EV Potential Problems

11. R R • Confound • Hard to detect the effect of NRexpbecausethe effect looks like it could be from NRexpbut could be due to the NRother NR other NR exp Difference Detector Experiment can detect an effect, but can’t tell where it is from Confounding

12. R R R R • Confound • Hard to detect the effect of NRexpbecausethe effect looks like it could be from NRexpbut could be due to the NRother These two situations look the same NR other NR exp NR other Difference Detector Difference Detector There is an effect of the IV There is not an effect of the IV Confounding

13. Excessive random variability • If experimental control procedures are not applied • Then R component of data will be excessively large, and may make NRexp undetectable Potential Problems

14. NR R R exp • If R is large relative to NRexp then detecting a difference may be difficult Difference Detector Experiment can’t detect the effect of the treatment Excessive random variability

15. NR R R exp • But if we reduce the size of NRother and R relative to NRexp then detecting gets easier • So try to minimize this by using good measures of DV, good manipulations of IV, etc. Difference Detector Our experiment can detect the effect of the treatment Reduced random variability

16. How do we introduce control? • Methods of Experimental Control • Constancy/Randomization • Comparison • Production Controlling Variability

17. Constancy/Randomization • If there is a variable that may be related to the DV that you can’t (or don’t want to) manipulate • Control variable: hold it constant • Random variable: let it vary randomly across all of the experimental conditions Methods of Controlling Variability

18. Comparison • An experiment always makes a comparison, so it must have at least two groups • Sometimes there are control groups • This is often the absence of the treatment Training group No training (Control) group • Without control groups if is harder to see what is really happening in the experiment • It is easier to be swayed by plausibility or inappropriate comparisons • Useful for eliminating potential confounds Methods of Controlling Variability

19. Comparison • An experiment always makes a comparison, so it must have at least two groups • Sometimes there are control groups • This is often the absence of the treatment • Sometimes there are a range of values of the IV 1 week of Training group 2 weeks of Training group 3 weeks of Training group Methods of Controlling Variability

20. Production • The experimenter selects the specific values of the Independent Variables 1 week of Training group 2 weeks of Training group 3 weeks of Training group • Need to do this carefully • Suppose that you don’t find a difference in the DV across your different groups • Is this because the IV and DV aren’t related? • Or is it because your levels of IV weren’t different enough Methods of Controlling Variability

21. So far we’ve covered a lot of the about details experiments generally • Now let’s consider some specific experimental designs. • Some bad (but common) designs • Some good designs • 1 Factor, two levels • 1 Factor, multi-levels • Between & within factors • Factorial (more than 1 factor) Experimental designs

22. Bad design example 1: Does standing close to somebody cause them to move? • “hmm… that’s an empirical question. Let’s see what happens if …” • So you stand closely to people and see how long before they move • Problem: no control group to establish the comparison group (this design is sometimes called “one-shot case study design”) Poorly designed experiments

23. Bad design example 2: • Testing the effectiveness of a stop smoking relaxation program • The participants choose which group (relaxation or no program) to be in Poorly designed experiments

24. Random Assignment • Bad design example 2: Non-equivalent control groups Independent Variable Dependent Variable Self Assignment Training group Measure participants No training (Control) group Measure Problem: selection bias for the two groups, need to do random assignment to groups Poorly designed experiments

25. Bad design example 3:Does a relaxation program decrease the urge to smoke? • Pretest desire level – give relaxation program – posttest desire to smoke Poorly designed experiments

26. Pre-test No Training group Post-test Measure • Bad design example 3: One group pretest-posttest design Dependent Variable Independent Variable Dependent Variable participants Pre-test Training group Post-test Measure Add another factor • Problems include: history, maturation, testing, and more Poorly designed experiments

27. Good design example • How does anxiety level affect test performance? • Two groups take the same test • Grp1 (moderate anxiety group): 5 min lecture on the importance of good grades for success • Grp2 (low anxiety group): 5 min lecture on how good grades don’t matter, just trying is good enough • 1 Factor (Independent variable), two levels • Basically you want to compare two treatments (conditions) • The statistics are pretty easy, a t-test 1 factor - 2 levels

28. Dependent Variable Random Assignment Anxiety Low Test participants Moderate Test • Good design example • How does anxiety level affect test performance? 1 factor - 2 levels

29. One factor Use a t-test to see if these points are statistically different test performance low moderate low moderate anxiety Two levels • Good design example • How does anxiety level affect test performance? anxiety 60 80 Observed difference between conditions T-test = Difference expected by chance 1 factor - 2 levels

30. Advantages: • Simple, relatively easy to interpret the results • Is the independent variable worth studying? • If no effect, then usually don’t bother with a more complex design • Sometimes two levels is all you need • One theory predicts one pattern and another predicts a different pattern 1 factor - 2 levels

31. Interpolation What happens within of the ranges that you test? test performance low moderate anxiety • Disadvantages: • “True” shape of the function is hard to see • Interpolation and Extrapolation are not a good idea 1 factor - 2 levels

32. Extrapolation What happens outside of the ranges that you test? test performance low moderate anxiety high • Disadvantages: • “True” shape of the function is hard to see • Interpolation and Extrapolation are not a good idea 1 factor - 2 levels

33. For more complex theories you will typically need more complex designs (more than two levels of one IV) • 1 factor - more than two levels • Basically you want to compare more than two conditions • The statistics are a little more difficult, an ANOVA (Analysis of Variance) 1 Factor - multilevel experiments

34. Grp3 (high anxiety group): 5 min lecture on how the students must pass this test to pass the course • Good design example (similar to earlier ex.) • How does anxiety level affect test performance? • Two groups take the same test • Grp1 (moderate anxiety group): 5 min lecture on the importance of good grades for success • Grp2 (low anxiety group): 5 min lecture on how good grades don’t matter, just trying is good enough 1 Factor - multilevel experiments

35. Random Assignment Dependent Variable Anxiety Low Test participants Moderate Test High Test 1 factor - 3 levels

36. anxiety mod high low test performance 60 80 low mod high anxiety 60 1 Factor - multilevel experiments

37. Advantages • Gives a better picture of the relationship (function) • Generally, the more levels you have, the less you have to worry about your range of the independent variable 1 Factor - multilevel experiments

38. 2 levels 3 levels testperformance test performance low mod high low moderate anxiety anxiety Relationship between Anxiety and Performance

39. Disadvantages • Needs more resources (participants and/or stimuli) • Requires more complex statistical analysis (analysis of variance and pair-wise comparisons) 1 Factor - multilevel experiments

40. The ANOVA just tells you that not all of the groups are equal. • If this is your conclusion (you get a “significant ANOVA”) then you should do further tests to see where the differences are • High vs. Low • High vs. Moderate • Low vs. Moderate Pair-wise comparisons