Lecture 3

1. Lecture 3 Theory and Measurement: Causation, Validity and Reliability

2. Assignment 1 • Stating a research problem. • Providing a short justification for the problem. • Providing a few hypotheses that come from the research problem.

3. Today’s Lecture • Discussion of Causation • Discussion of Validity • Discussion of Reliability • Time permitting, Issues in data preparation and error-checking

4. Reminder from Lectures 1 and 2: Causation versus Correlation • Correlation: • Non-directional relationship between two variables. • Increase in X associated with Increase in Y, but could also be stated as Increase in Y associated with increase in X. • Causation: • Directional relationship between at least two variables. • Increase in X leads to increase in Y, but reverse may or may not be true.

5. Causation • Key to causation is directionality • Must be able to establish directionality either theoretically, methodologically, or both.

6. Porter’s (1997) Three Criteria for Cause • Independent variable must precede the dependent variable. • Independent variable must be related to the dependent variable. • There must be no third variable that could explain why the independent variable is related to the dependent variable.

7. Example: Age and Income • We know they are correlated, so does age cause income to increase? • We know that Income cannot ‘cause’ age. • They certainly seem related and the direction seems clear… so is it not clear that age causes income to increase? • “Third Variable” problem: age related to education, job experience, other factors.

8. Five Approaches to Quantitative Research and Implications for Causality • Descriptive • Associational • Comparative • Quasi-Experimental • Randomized Experimental

9. Research types and causality: Descriptive • Descriptive • Summarize data • Statistics: histograms, means, percentages • Cannot show causality

10. Research types and causality: Associational • Associational • Only to relate variables • Predictions only made to show that a relationship exists • Statistics: Correlation, Multiple Regression • To some degree, regression can partially be used to infer causality

11. Research types and causality: Comparative • Comparative • Compares two or more groups • Looking for difference between groups • Statistics: t-tests, ANOVA (inferential statistics) • Not well suited for establishing cause b/c it does not meet Porter’s (1997) 3rd condition (extraneous variables)

12. Research types and causality: Quasi-experimental • Quasi-experimental • Compares groups • ‘quasi-experimental’ b/c it does not have random assignment to groups. • Can examine causality • Statistics: t-tests, ANOVA (inferential statistics)

13. Research types and causality: randomized experimental • Randomized experiment • To determine causes • Compares groups • Has random assignment to groups • Best way to determine exact causes • Statistics: t-tests, ANOVA (inferential statistics)

14. In Summary… • You just need to know that the type of research that you do will affect your ability to describe causality. • Whenever possible, choose a research method that will allow you to have the most explanatory power.

15. Validity in Research:The ‘quality’ or merit of research

16. Validity: Internal, External, Measurement • Internal Validity: “the approximate validity with which we can infer that a relationship is causal” (Cook and Campbell 1979). • External Validity: “external validity asks the question of generalizability: to what populations, settings, treatment variables, and measurement variables can this effect be generalized?” (Campbell and Stanley 1966). • Measurement Validity: Do our measures capture what we want them to capture?

17. Internal Validity • Two major threats to internal validity (is our study causal)?: • Equivalence of groups on participant characteristics • Control of extraneous experience or environment variables.

18. Equivalence of Groups • If looking at a specific cause (X affects Y), then the groups must not vary significantly on other key variables. • Example: Looking at the effect of computer use on intelligence. • But what if computer users and non-computer users differ on employment, age, education, etc?

19. Control of Extraneous Experience or Environment Variables • If looking at a specific cause (X affects Y), then one or more groups cannot receive unknown stimuli or information that could affect outcome. • Problem is particularly troublesome if it affects groups differentially. • Example: Study of two classrooms, one with information technology and one without such technology. • What if one of the classes also has teaching assistants who help the students?

20. Other types of Internal Validity Problems • Statistical Regression • Because of statistical variation, some individuals may be placed in wrong group (extremes regress to mean) • Experimental Mortality • Some individuals ‘leave’ study– if this is systematic for certain groups, it’s a problem. • Selection • Process for assigning to different groups. • Interactions with Participant Assignment • Biases in assignment to groups can also have interactions between groups (i.e., environmental factors that differentially affect certain individuals who were not randomly assigned to groups).

21. External Validity • How generalizable is a given study? • Two major types: • Population external validity • Ecological external validity

22. Population External Validity • Population External Validity: • Does the actual sample of participants represent the theoretical or target population? • To evaluate, you must know: • The theoretical population • The accessible population • The sampling design • The selected sample • The actual sample who complete study

23. Ecological External Validity • Ecological External Validity • Are the conditions, settings, procedures, questions, etc representative of real life? • Often, ecological external validity in competition with experimental controls that attempt to isolate specific variables. • Example: • Study of sharing behavior in P2P-like systems (Cheshire 2005)

24. Why care about external validity? • 1930’s Literary Digest poll: • Franklin Roosevelt predicted to lose the 1936 presidential election by a landslide. • Oops… he won by a landslide. • How could this happen? • Sample was selected from automobile registrations, telephone directories…during the middle of the Great Depression.

25. Related point: outliers in sample • What is the best undergraduate major if you want a high income (UNC-Chapel Hill survey)? • Geography was #1 • Maybe not time to switch majors just yet… • One outlier, Michael Jordan, accounted for the huge skew in average salaries for graduates (he makes $80 million/year) • Key Point: you have to try and make every effort to make your sample generalizable to the population of interest. • Non-representative samples will lead to inaccurate conclusions!!!

26. Measurement Validity • Deals with whether the variables are appropriately defined and representative of the concepts or constructs under investigation. Also called construct validity. • Examples: • How do you measure life happiness? • How do you measure technical proficiency? • How do you measure one’s social network?

27. Example of Measurement Validity Problem • Operational definition of ‘supervision’ is defined as a supervisor being 10 feet or less from a worker (example from Cook and Campbell 1979) • Problem: the way that supervision is defined, it may be relevant to the construct of ‘stress’ rather than just “supervision”.

28. Validity: Summary • Internal Validity: • Has the causal link between our concepts (or variables) been established? • External Validity: • Is the study generalizable, and to what group(s)? • Measurement Validity: • Do our measures capture what we want them to capture?

29. Reliability in Research

30. Reliability • Reliability deals with the consistency of your research instrument (i.e., survey questions, experimental manipulations, etc.)

31. Reliability • Are the findings (or a specific measure) consistent if you were to do the study over again? • A study can be reliable, but not valid. Furthermore, it cannot be valid unless it is reliable. • Thus, reliability is absolutely required. Validity is equally important, but the degree of validity (such as external validity) may not be very high depending on the nature of the study.

32. Reliability: the problem of error • Error is the difference between the observed score and the ‘true’ score. • Random error occurs: • Due to observers… • Due to individual variation (age, mood, etc) • Due to inconsistent situations during data collection (i.e., survey on patriotism after 9/11)

33. Methods of Measuring Reliability • Split-half or item performance • Analyze half of survey/instrument and compare to overall analysis to see if it is consistent. • Cronbach’s alpha is a related and common way to measure reliability (correlating performance on each item with overall score)

34. Three More Methods of Measuring Reliability • Test-retest • Administering test to same group at different times, correlate the two scores. • Multiple or Parallel forms • Mixing same items on a survey and giving to same group twice. • Inter-rater reliability • Agreement between different interviewers or coders on same subjects/responses.

35. Summary: Reliability • Basically, reliability just deals with the consistency of your measures. If you can show that they are consistent, then you have this covered.

36. Class Survey: Data Preparation and Error-Checking