Research Designs: the Good, the Bad, and the Ugly

1. Research Designs: the Good, the Bad, and the Ugly Dahlia Forde Graduate Research Assistant Faculty Center

2. Overview Introduction Basic Terms Steps to Conducting SoTL Research Procedures for Designing Research Types of Research Threats to Validity Types of Research Designs Conclusions

3. Introduction Focus will be on: Research designs that are common in SoTL research Quantitative research

4. What is SoTL? The Scholarship of Teaching and Learning (SoTL) is the investigation of problems in teaching and learning. Empirical research is used to study classroom practices Results are communicated to peers

5. What is Research Design? Research design is the structure or blueprint of the research A plan developed in advance specifying the methods and procedures of collecting and analyzing the information/data needed to address the research question(s)

6. Steps in Conducting SoTL Research State problem related to teaching and learning Develop initial research question(s) Review literature Develop research question(s) and/or hypotheses Design Study Determine statistical methods Get IRB approval Collect data Analyze data and interpret results Report findings in reference to research questions and hypotheses

7. Steps for Research Design Review research questions and/or hypotheses Identify and operationally define variables/constructs Determine population and then sampling strategy Select research design Select method of analysis

8. Types of Research Qualitative vs. Quantitative Cross sectional vs. Longitudinal Experimental vs. Non-Experimental Type of Data What you are doing with data (manipulating it or measuring it or both) Time frame of data collectionType of Data What you are doing with data (manipulating it or measuring it or both) Time frame of data collection

9. Qualitative vs. Quantitative Qualitative Research Used during earlier phases of research Data are in verbal, text, or visual form Data are �rich�, time consuming and less able to be generalized Subjective: individuals� interpretation of written or verbal data

10. Qualitative vs. Quantitative One type of research is not better than the other, each serves its own purpose. Some research questions are best answered by using both methods (aka mixed methods)

11. Qualitative vs. Quantitative Qualitative Methods Course portfolio and other reflection of analysis Interviews and focus groups Observation Content analysis Questionnaires

12. Cross-sectional vs. Longitudinal Time Frame Cross-sectional Data collected at one point in time Longitudinal Data collected at more that one point in time Example: pre- post-test during one semester Example: assessing the impact of an intervention over time (from year to year) by assessing retention and grade-point average.

13. Non-Experimental vs. Experimental Non-Experimental Measurement of variables No random assignment Weak causal inference Experimental Manipulation and measurement of variables Random assignment Strong causal inference

14. Threats to Validity Internal threats to validity Construct threats to validity External threats to validity Statistical conclusion validity SoTL research can involve experimental and quasi experimental research There is manipulation of IV, but the difference is random assignment does not occur for quasi expSoTL research can involve experimental and quasi experimental research There is manipulation of IV, but the difference is random assignment does not occur for quasi exp

15. Threats to Internal Validity Ambiguous Temporal Precedence Selection History Maturation Regression Mortality Testing Instrumentation Interactions with multiple threats

16. Social Interaction Threats Diffusion or Imitation of Treatment Compensatory Equalization Compensatory Rivalry Resentful Demoralization

17. Threats to External Validity

18. Research Designs Post-test Only One Group Pre- Post-test Static Group Comparison Quasi-Experimental (Pre- Post-tests w/ two groups) True Experiment Solomon Four Design Passive Observational / Correlational

19. Research Design: Basic Notation X = Treatment O1 = First Observation O2 = Second Observation R = Randomization

20. Post-test Only Design Research Question: Are collaborative projects effective for honors students? Hypothesis: Honors students engaging in team projects will be satisfied with the course. Design: Post-test Only Design or One Shot Case Study

21. Post-test Only Design Statistical Method: One sample t-test Variables needed: continuous scores from one group Threats to Validity: History, Maturation, Selection, Mortality, Selection by Treatment Problems: No control; little scientific value; no meaningful comparisons Recommendation(s): Do not use. BAD3 design.

22. One Group Pre- & Post-test Design Research Question: Are collaborative projects effective for honors students? Hypothesis: Honors students engaging in team projects will show an increase in satisfaction with the course. Design: One Group Pre- and Post-test Design

23. One Group Pre- & Post-test Design Statistical Method: Paired samples t-test Variables needed: one categorical IV (e.g. Time); and one continuous variable measured on two different occasions or under two conditions Threats to Validity: History, Maturation, Testing, Instrumentation, Statistical Regression, Interactions: Testing x Maturation, Testing x Intervention, Selection x Intervention Recommendation(s): Use when there is no other comparison group; The group is compared with itself.

24. Static Group Comparison Design Research Question: Does relative grading influence student collaboration? Hypothesis: Students in a course with relative grading will meet in study groups fewer times per week than students in a course with standard (absolute) grading. Design: Two Group Post-test Design or the Static Group Comparison

25. Static Group Comparison Design Statistical Method: Independent samples t-test Variables needed: one categorical IV; and one continuous DV Threats to Validity: History, Maturation, Selection, Mortality, Interactions: Selection x Maturation, Selection x Intervention Recommendation(s): Use when pre-test and randomization is not possible

26. Quasi-Experiment Research Question: Does semantic association improve memory recall? Hypothesis: Students who receive meaningful pictures paired with written concepts will correctly recall more concepts than students who only receive written concepts. Design: Quasi Experiment IV is manipulated Students are not randomly assigned IV: discrete (semantic association vs. no association) DV: continuous (recall score)

27. Quasi-Experiment Statistical Method: Mixed between-within subjects analysis of variance (ANOVA) Variables needed: two categorical IV�s [one between group and one within group (i.e., TIME)], and one continuous DV. Threats to Validity: Testing, Maturation, History, Interactions: Selection by Treatment, Maturation by Treatment Recommendation(s): Collect demographic data to examine whether or not the two groups are similar especially on variables that my influence the DV.

28. True Experiment Research Question: Will students, in online courses, perform better on instructor monitored collaborative work than non-monitored collaborative work? Hypothesis: In an online course, students whose collaborative work is being monitored will have higher team project scores than students whose collaborative work is not being monitored. Design: True Experiment IV is manipulated Participants are randomly assigned to conditions

29. True Experiment Statistical Method: Mixed between-within subjects analysis of variance (ANOVA) Variables needed: two categorical IV�s [one between group and one within group (i.e., TIME)], and one continuous DV. Threats to Validity: History, Maturation, Testing, Instrumentation, Statistical Regression, Mortality, Interactions: Testing x Intervention, Selection x Treatment Recommendation(s): Good control; use parallel forms of pre- and post-test

30. Solomon Four Design Research Question: Will students, in online courses, perform better on instructor monitored collaborative work than non-monitored collaborative work? Hypothesis: Will students, in online courses, perform better on instructor monitored collaborative work than non-monitored collaborative work? Design: Solomon Four Design IV is manipulated Participants are randomly assigned to conditions There are control groups Groups with and without pre-tests

31. Solomon Four Design Statistical Method: 2x2 analysis of variance (ANOVA) design pre-tested against un-pretested Variables needed: two categorical IV�s; and one continuous DV. Threats to Validity: No threat to internal validity Recommendation(s): Great control Great internal validity

32. Passive Observational / Correlational Research Question: Is self-efficacy related to classroom performance? Hypothesis: Students with high scores on self-efficacy will have higher final exam scores than students with low scores on self-efficacy. Design: Passive Observational / Correlational study Variables are continuous and not manipulated

33. Passive Observational / Correlational Statistical Method: Correlation Variable(s) needed: two continuous variables Threats to Validity: All internal validity threats apply Recommendation(s): Cannot make causal inferences

34. Summary Introduction Basic Terms Types of Research Threats to Validity Types of Research Designs Conclusions

35. Questions ?

36. Thank you!