COMM 301: Empirical Research in Communication

1. COMM 301:Empirical Research in Communication Kwan M Lee Lecture 7

2. Remember how I started Lect4 (Measurement validity and reliability)?

3. Internal vs. External Validityof a Study (from Lect 4) • External validity • Is it generalizable? • Sampling (lect5) and replication • Internal validity • Is it accurate? • Can be influenced by measurement validity, measurement reliability, and other factors (E.g., history; maturation; sensitization; experiment demand;…..) • Lect 4 focuses on measurement validity and reliability.

4. Today’s Topic – More on Validity • Internal validity • External validity

5. Internal validity • Internal validity • accuracy of a study’s results based on its design and execution • influenced by • measurement reliability and validity • procedures used in data collection • a set of possible threats to validity

6. Factors Influencing Internal Validity • Measurement reliability and validity • Study procedures • Standardization of study settings and research procedures. • e.g. use of scripts for each researcher, each stage of the study • Valid and working manipulations and treatments • e.g. negative campaign advertising is indeed negative • Set of possible threats to validity

7. Possible Threats to Internal Validity • Maturation • natural physical and psychological change in subjects due to passing of time, independent of intervention • more problematic for studies spanning across time • e.g. effect measures very long after exposure

8. Possible Threats to Internal Validity (cont.) • Attrition • Elements (including subjects) initially included in the study lost as study progresses • e.g. human subjects withdraw because of fatigue due to long questionnaire, lack of interest, illness, so on • e.g. tape recordings destroyed

9. Possible Threats to Internal Validity (cont.) • Sensitization • participants’ experience with research procedures • e.g. similar or same questions (in pretest post test situations) • e.g. previous exposures or interventions (sequencing effects  counterbalancing)

10. Possible Threats to Internal Validity (cont.) • Inter-subject interaction • multiple subjects come into contact, and share information about the study • e.g., sharing info about experiment deception • Hawthorne effect • Subjects behave differently because they know that they are being observed • E.g. lightning and productivity

11. Possible Threats to Internal Validity (cont.) • Regression to the mean • natural movement towards the average • problematic when we select subjects based on subjects being far from average • e.g. students who performed worst (lowest 10%) in pretest and their natural enhancement in post-test scores

12. Possible Threats to Internal Validity (cont.) • Regression to the mean (cont.) • Why • Non-probability sample • Less than perfect correlation between two measures • The more extreme the sample group and/or the less correlated the two variables  the greater the amount of changes due to regression to the mean.

13. Possible Threats to Internal Validity (cont.) • Researcher presence • Personal attribute effect • researcher’s physical characteristics (gender, ethnicity, dress, tone … ) affects the data collected • e.g. male versus female interviewer on sexual harassment • remedy: important to try to match researcher with subjects on important characteristics

14. Possible Threats to Internal Validity (cont.) • Researcher presence (cont.) • Unintentional expectations • researcher inadvertently gives cues about what kind of data is desired • subject may either provide desired data or provide undesired data • remedy: use double-blind procedure • use of a confederate who does not know about the purpose of the study

15. Possible Threats to Internal Validity (cont.) • History • events participants experienced during the course of a study • events naturally occurring, outside control of the researchers • e.g., long term study of the effect of Internet on loneliness --> if a tragedy happens during the course of a study, it will affects the study outcome.

16. A review of factors influencing the internal validity of a study • Internal Validity is influenced by • measurement reliability and validity • Check Lect 4 • procedures used in data collection • Standardization of the study settings and research procedures • Validity of manipulations and treatments • a set of possible threats to validity • Maturation • Attrition • Sensitization • Inter-subject interaction • Hawthorne effect • Regression to the mean • Researcher presence • history

17. Why do we need to know various threats to internal validity? • When conducting a study • Try to control factors affecting internal validity • At least, know about the weakness of your study • When consuming a research report • Scrutinize the study against the factors you have learned so far

18. External validity • External validity • accuracy with which the study’s findings can be generalized • influenced by • sampling • ecological isomorphism • replication

19. Factors Influencing External Validity • Sampling • key issue: representativeness • Ecological isomorphism • similarities between the research setting and the real world setting • e.g. studying media exposure

20. Factors Influencing External Validity (cont.) • Replication • Do similar studies provide similar results? • literal replication • exact duplication in problem, methods, procedures, measurement, population • different sample from same population • general replication • same problem, method • procedures, measurements, population modified

21. Factors Influencing External Validity (cont.) • Replication (cont.) • Triangulation • same general problem, but different method • Ideal research strategy: Methodological Pluralism (Hovland) • e.g. study conflicts in close interpersonal relationships using experimental design and surveys