Objectives

1. Slides to accompany Weathington, Cunningham & Pittenger (2010), Chapter 4: An Overview of Empirical Methods

2. Objectives • Internal, statistical conclusion, and external validity • Empirical methods • Intact groups and quasi-experimental designs • Surveys • Correlational studies • Single-N methods • Meta-analysis

3. Internal Validity • Shown by the degree to which a study rules out alt. explanations for IV  DV • Requires ruling out alternative explanations • Threats include sources of confounding variables • 4 general categories

4. Threats to Internal Validity • Unintended sequence of events • Carryover effects: drug at Time 1 hurts performance at Time 2 (but the drug is not what we wanted to test) • Maturation: Changes in answers between 6 and 10 year olds may be due to normal learning rather than a reading intervention • Intervening events: being burglarized may change your response to a social psychology experiment involving eye witnesses

5. Threats to Internal Validity • Nonequivalent groups • Confounds interpretation of cause and effect between IV and DV • Can be caused by: • Non-random sampling • Mortality/attrition • Subject characteristics (variables)

6. Threats to Internal Validity • Measurement errors • Non-valid test • Low reliability of measurement • Ceiling and floor effects • Regression to the mean • Ambiguity of cause and effect • Which came first, X or Y?

7. Statistical Conclusion Validity • Were the proper statistical or analytical methods used when studying the data? • “Proper” = best allowing the researcher to: • Demonstrate relationship between IV and DV • Identify the strength of this relationship

8. Threats to Statistical Conclusion Validity • Low statistical power: increases risk of missing an effect that really exists • Violating assumptions of tests: no statistical tests are perfect in all research situations; you need to know your “tools” • Unreliability in measurement and setting: inconsistencies in the measurement process make it impossible for you to draw valid inferences from the statistics

9. External Validity • Do our findings/results generalize beyond our sample? • More likely if representative sample • Can we generalize our findings to the population? • Can we generalize our conclusions from one population to another?

10. Internal vs. External Validity

11. Threats to External Validity • NOT always just the “lab setting” • Participant recruitment • How + who you select to study matters • Need to be as representative as possible • May require replication, extension studies

12. Threats to External Validity • Situation effects • Where you do the study matters • Control for what you can and consider replicating in different settings • History effects • Be aware that phenomena may change over time

13. True Experiment • Best method for testing cause and effect • “Easiest” control for internal validity threats • Not always a practical/ethical option • You know it is a true experiment if: • The IV can be controlled/manipulated • Random assignment to conditions occurs • Control conditions can be created

14. Sampling frame Assuming random assignment into groups, differences Random assignment among the groups at this stage are due to Group 1 Group 2 Group 3 random effects n = 10 n = 10 n = 10 Separate conditions controlled by the Treatment for Treatment for Treatment for researcher (different Group 1 Group 2 Group 3 levels of IV) Differences among groups due to Results for Results for Results for random effects + Group 1 Group 2 Group 2 effect of treatment (level of IV) Nonrandom differences among the groups in terms of the measured DV leads us to conclude that the manipulations of the IV may have caused those differences True Experiment

15. Intact Groups Design • No random assignment possible • Multiple samples (by subject variables), from multiple populations • Cannot establish cause and effect • Unknown 3rd variable and temporal order • Can compare differences across samples

16. Intact Groups Design

17. Quasi-Experimental Design • No random assignment; grouping by some other factor • An IV is manipulated • One group is treated as a “control”, while the other is exposed to the manipulated IV • Still problem with unknown 3rd variable and temporal order

18. Quasi-experimental Design

19. How does the true experiment differ from the intact groups and quasi-experiment design?

20. Surveys • For estimating population parameters • Good for large-scale data collection • Quick and inexpensive • “Bad” because of respondent error • Honesty and personal bias

21. Correlational Study • Usually to estimate population parameters • Often data from surveys • Good for initial understanding and “prediction” of complex behaviors • Bad at supporting cause and effect • Unknown 3rd variable • Temporal order issues

22. Single-N Methods • Sometimes better to focus in-depth on one or a few participants • Single-participant experiment • Case study • Good if IV and situational variables are well-controlled • Bad for generalizability (potentially) and also because of participant bias/error

23. Meta-analysis • Analysis of multiple outcomes from multiple studies • Good because takes advantage of more representative sampling of participants and measures/methods • Bad because depends on which studies are entered • Principle of GI, GO

24. What is Next? • **instructor to provide details