Research Design

1. Research Design Week 4 Laura Christiansen

2. Introduction Structure of research Elements of design Group Assignments Measures/Observations Programs/Treatments

3. Design Notation Grouping Notations R = random N = non-equivalent C = cutoff O = observations X = program/treatment Blank or +/- notation N O1 X O2 N O1 O2 C O X O C O O

4. Research Design Types Experimental Random Assignment � probabilistically equivalent Strongest internal validity Quasi-Experimental Multiple measures Multiple groups/control group

5. Experimental Designs Two-Group Signal Enhancers Factorial Designs Noise Reducers Randomized Block Covariance Hybrid Experimental Solomon Four-Group Switching Replications

6. Two-Group Experimental Designs No necessary pretests Probabilistically equivalent Strong internal validity against: Single-group, (most) multiple-group Weak internal validity against: Social interaction, selection mortality

7. Factorial Designs Factors: Major independent variables Levels: Subdivisions of factors Notation example: 5x6 # of terms determines # of factors: 2 # of values determines levels: one factor has 5, the other has 6 Multiply to determine necessary groups: 30

8. Factorial Designs Cont. Null Case No effect, no graphical slope Main Effects Consistent statistical difference between levels Lines still parallel Interaction Effects One factor depends on level of another Lines not parallel

9. Factorial Design Cont. Increasing number of factors Difficult to graph Increased number of groups Incomplete factorial designs Useful with control groups Leave out combinations

10. Randomized Block Separate into homogenous blocks Less variability/noise Pool estimates across blocks Data analysis strategy

11. Covariance Adjust post-test data for pre-test variability �Covary� post-test with pre-test Noise reduction Removes pre-post relationship

12. Solomon-Four Group Deals with testing threat Can view as 2x2 factorial Analyze difference between control and program Analyze difference between pre-test and non-pretest

13. Switching Replications 2 groups, 3 waves of measurement Helps mitigate social threats

14. Quasi-Experimental Designs Non-Equivalent Groups Regression Discontinuity Proxy Pretest Separate Pre-Post Samples Double Pretest Switching Replications Non-Equivalent Dependent Variables Pattern Matching Regression Point Displacement Interrupted Time Series

15. Non-Equivalent Groups N O X O N O O Groups are selected based on similarity Threat of selection Concerns regarding internal validity in analysis Popular in social research

16. Regression-Discontinuity C O X O C O O Assignment based on cutoff scores Targeted programs Pre-test/post-test measure equivalence not required Assumes no natural discontinuity at cutoff Less common in social research Comparable to randomized experimental IV for causal hypotheses Ethical benefits in some fields

17. Proxy-Pretest N O1 X O2 N O1 O2 Measure of O1 occurs after X �Proxy� variable estimate Two types: �Recollection� � ask for approximation �Archived� � construct approximation Do not plan to use this

18. Separate Pre-Post Samples N1 O N1 X O N2 O N2 O Pre-test and post-test data come from different groups Nonequivalence concerns Variants include random selection within non-equivalent groups

19. Double Pretest N O O X O N O O O Stronger variant of Non-Equivalence Groups Design Detection of selection threats Detection of selection-maturation Also called �dry run� Simulates control group

20. Switching Replications 2 groups, 3 waves of measurement Equivalent to experimental variant Ethical benefits among quasi-experiments N O X O O N O O X O

21. Non-Equivalent Dependent Variables Targeted at specific outcome Single group divided in two Pre/post O2 tests act as control and gauge of maturity Minimize risk �control� being impacted by X Weak internal validity

22. Pattern Matching NEDV Variant of Non-Equivalent Dependent Variable design Incorporate multiple subgroups of single non-equivalence group Include theorization on affect of program Not concerned about lone impact as with NEDV Order of effects

23. Regression Point Displacement Compares test group to heterogeneous control groups Useful when dealing with multiple factors Judge post-test based on relation to regression line of control cases

24. Interrupted Time Series Weak on internal validity Can be combined with other designs Ex: Interrupted Time Series w/ Comparison Group Strengthens some benefits of Double Pretest