PhD Research Seminar Series: Valid Research Designs

1. PhD Research Seminar Series:Valid Research Designs Dr. K. A. Korb University of Jos

2. Outline • Threats to valid Research Design • Construct Validity • Internal Validity • External Validity • Reliability Dr. K. A. Korb University of Jos

3. Reliability: Consistency of results Reliable Unreliable Dr. K. A. Korb University of Jos

4. Validity: Measuring what is supposed to be measured Valid Invalid Dr. K. A. Korb University of Jos

5. Research Example • The effect of regular leisure reading on reading achievement in primary school. • Students in a primary school will be randomly assigned to either a treatment or a control group. • The treatment group will spend 10 minutes everyday reading a book with an adult. • In an attempt to keep the treatment and control groups as similar as possible, the control group will also spend 10 minutes with an adult everyday, but will instead do math problems. Dr. K. A. Korb University of Jos

6. Research Example IV: Regular Leisure Reading DV: Reading Achievement Experimental Group Reading Achievement Read Book Every Day with Adult Reading Achievement Control Group Do Maths Every Day with Adult Reading Achievement Reading Achievement Dr. K. A. Korb University of Jos

7. Construct Validity • Construct Validity: Establishing valid operational measures for the concepts being studied • This relates to: • Independent Variable • Dependent Variable • Review: • Construct Definition: General explanation of the construct by relating it to other constructs • Operational Definition: Statement of specifically how the construct will be measured or implemented in your study Dr. K. A. Korb University of Jos

8. Review:Construct vs. Operational Definitions Construct Operational Definition WAEC Scores Final Course Exam AcademicAchievement NECO Scores State Exams Self-Report of Parents’ Yearly Income Economic Status Free/Reduced Lunch Eligibility Dr. K. A. Korb University of Jos

9. Research ExampleConstruct validity focuses on: IV: Regular Reading DV: Reading Achievement Experimental Group Reading Achievement Read a Book Every Day Reading Achievement Reading Achievement Reading Achievement Control Group Dr. K. A. Korb University of Jos

10. Construct Validity • When you develop your research project, you start with a theory. • For example, regular leisure reading will lead to increased reading achievement scores. • Then you operationalize your theory into your research study. • You translate “regular leisure reading” into how it will be implemented in your study – reading a book with an adult everyday for 10 minutes. • You translate “reading achievement scores” into how it will be measured in your study – classroom exam scores. • When you finish collecting data, you hope to be able to move back to theory. • You want to be able to say that not only did reading a book for 10 minutes everyday with an adult lead to higher classroom exam scores on reading achievement, but also that regular leisure reading will lead to higher reading achievement scores in general. • However, in order to do that, you must have construct valid measures of your independent and dependent variables. Dr. K. A. Korb University of Jos

11. Research Example IV: Regular Reading DV: Reading Achievement Theory: Scores on reading section of classroom exams Read book every day with an adult for 10 min. Research Study: IV: Regular Reading DV: Reading Achievement Theory: Dr. K. A. Korb University of Jos

12. Construct Validity • Enables you to generalize from the treatment and measures in your study to the general constructs of your treatment and measures • Threats to Construct Validity • Inadequate explanation of constructs in your paper • Flawed matching of operationalization to constructs • The measures do not align with the constructs they were designed to measure. Dr. K. A. Korb University of Jos

13. Internal Validity • Internal validity: Extent to which variables other than the treatment provide plausible explanations to the experimental results • Only relevant for experimental designs • Primary consideration for interventions • Extraneous variable: Any variable other than treatment variable that if not controlled can affect the experimental outcome • Hold constant or eliminate all extraneous variables that might affect the posttest • Goal of Research Design: Create set of conditions so any observed changes in your dependent variable can be attributed to experimental treatment instead of extraneous variables • Do variables other than the treatment provide plausible explanations to the experimental results? Dr. K. A. Korb University of Jos

14. Research Example Maturation IV: Regular Reading DV: Reading Achievement Interest Experimental Group Reading Achievement Read a Book Every Day Reading Achievement Reading Achievement Reading Achievement Control Group Reading Ability Adult Attention Dr. K. A. Korb University of Jos

15. Threats to Internal Validity • History: If treatment lasts over time, other events may influence dependent variable • To prevent: Keep the control group equivalent in all aspects but treatment • Maturation: Physical or psychological changes may influence the dependent variable • To prevent: Use a control group • Testing: Giving a pretest may increase performance on the post-test • To prevent: Don’t use a pre-test OR Use different pre- and post-tests • Instrumentation: Nature of measuring instrument has changed between pre- and post-test • To prevent: Do not use a pre-test OR Use the same for pre- and post-test • Selection Bias: Treatment and control groups are different on an important extraneous variable • To prevent: Random assignment OR Use a pre-test • NOTE: Many of these preventions contradict. You must think about which threats are most relevant to your study and then develop a strategy for overcoming the most important threats. Dr. K. A. Korb University of Jos

16. External Validity • External Validity: Establishing the group of people to which the research findings can be generalized Dr. K. A. Korb University of Jos

17. Research Example Population Experimental Group Reading Achievement Read a Book Every Day Reading Achievement Sample Control Group Reading Achievement Reading Achievement Dr. K. A. Korb University of Jos

18. Types of External Validity • Population Validity: Generalizing the results from experimental sample to a defined population • To establish: Give a thorough explanation of the constitution of your sample and how your sample was selected. • Ecological Validity: Generalizing the results of a study from the artificial conditions created by researcher to real-life conditions • To establish: • Give an adequate description of treatment • Give an adequate description of the measure for the dependent variable • Threats to Ecological Validity: • Novelty: The treatment was different than typical treatment • Experimenter Effect: The particular person delivering the treatment affects results Dr. K. A. Korb University of Jos

19. Reliability • Reliability: Demonstrating that the operations of the study can be repeated with the same results • Conduct research as if someone were always looking over your shoulder • Report the procedure of your study in explicit detail Dr. K. A. Korb University of Jos

20. Other Issues in Valid Research Design • Experimenter Bias: Researcher’s expectations about the outcome of experiment influence participants’ response • To prevent: Use research assistants who are blind to the study (aka do not know the purpose of the study) • Treatment Fidelity: Extent to which treatment conditions are implemented according to the researcher’s specifications • To support: • Carefully train research assistants • Periodically check up on research assistants unexpectedly to determine if they are following guidelines • Strong Experimental Treatments: Developing a treatment that will make a robust effect on the dependent variable • To support: • Develop a thorough understand of your dependent variable and how it can be influenced. • Spend much thoughtful time developing your treatment. Dr. K. A. Korb University of Jos