The Theory of Sampling and Measurement

1. The Theory of Sampling and Measurement

2. Sampling • First step in implementing any research design is to create a sample. • We cannot study the theoretical population of all conceivable events (e.g., events that have not occurred), nor can we usually study all instances of actual events. We select some instances to study and not others. Those we include are our sample. • How our sample is selected is critical for external validity or generalizability.

3. Groups in Sampling Who do you want to generalize to?

4. Groups in Sampling The theoretical population

5. Groups in Sampling The theoretical population What population can you get access to?

6. Groups in Sampling The Theoretical Population The study population

7. Groups in Sampling The theoretical population The study population How can you get access to them?

8. Groups in Sampling The theoretical population The study population The sampling frame

9. Groups in Sampling The theoretical population The study population The sampling frame Who is in your study?

10. Groups in Sampling The theoretical population The study population The sampling frame The sample

11. Probability Sampling Simple random Stratified random Cluster or area random Non-Probability Sampling Accidental Modal instance Expert Snowball Case study (intentional selection) Types of Samples

12. Sample Sample Sample 5 5 5 0 0 0 5 5 5 0 0 0 3 . 0 3 . 2 3 . 4 3 . 6 3 . 8 4 . 0 4 . 2 4 . 4 3 . 0 3 . 2 3 . 4 3 . 6 3 . 8 4 . 0 4 . 2 4 . 4 3 . 0 3 . 2 3 . 4 3 . 6 3 . 8 4 . 0 4 . 2 4 . 4 1 5 1 0 5 0 3 . 0 3 . 2 3 . 4 3 . 6 3 . 8 4 . 0 4 . 2 4 . 4 The Sampling Distribution Average Average Average ...is the distribution of a statistic across an infinite number of samples. The sampling distribution...

13. Population Parameter The population has a mean of 3.75... 1 5 0 ...and a standard unit of .25. 1 0 0 Frequency 5 0 0 3 . 0 3 . 5 4 . 0 4 . 5 This means Self esteem About 64% of cases fall between 3.5 - 4.0. About 95% of cases fall between 3.25 - 4.25. about 99% of cases fall between 3.0 - 4.5

14. Sampling Distribution The population has a mean of 3.75. 1 5 0 1 0 0 Frequency 5 0 0 3 . 0 3 . 5 4 . 0 4 . 5 Self-esteem

15. Sampling Distribution The population has a mean of 3.75... 1 5 0 ...and a standard error of .25. 1 0 0 Frequency 5 0 0 3 . 0 3 . 5 4 . 0 4 . 5 Self-esteem

16. Inferring Population from Sample The sample has a mean of 3.75... 1 5 0 ...and a standard deviation of .25. 1 0 0 Frequency 5 0 0 3 . 0 3 . 5 4 . 0 4 . 5 This means Self esteem 64% chance true population mean falls between 3.5 - 4.0. 95% chance true population mean falls between 3.25 - 4.25. 99% chance true population mean falls between 3.0 - 4.5

17. Figure 3.4 Labor Repression and Growth in the Asian Cases, 1970-1981

18. Figure 3.5 Labor Repression and Growth in the Full Universe of Developing Countries,1970-1981

19. Measurement • Operationalization is the process of translating theoretical constructs into observable indicators. • Construct validity and reliability are the criteria we use to evaluate how well you have operationalized your concepts. • Both matter regardless of the level of measurement and whether you are using qualitative or quantitative indicators.

20. The Hierarchy of Levels Ratio Absolute zero Interval Distance is meaningful Ordinal Attributes can be ordered Nominal Attributes are only named; weakest

21. Nominal Measurement • The values “name” the attribute uniquely. • The name does not imply any ordering of the cases.

22. Ordinal Measurement When attributes can be rank-ordered… • Distances between attributes do not have any meaning.

23. Interval Measurement When distance between attributes has meaning, for example, temperature (in Fahrenheit) -- distance from 30-40°F is same as distance from 70-80°F • Note that ratios don’t make any sense -- 80°F is not twice as hot as 40°F.

24. Ratio Measurement • Has an absolute zero that is meaningful • Can construct a meaningful ratio (fraction), for example, number of clients in past six months

25. Construct Validity • Key problem is that we have abstract theoretical construct – power, democracy, development, corruption, etc. – that we can never observe directly. • Yet, to test propositions requires that we have some indicator for the construct – or at least have proxies that we can argue are capturing some attributes of the construct. • Our indicator is an analogy (to an analogy).

26. Assessing Construct Validity • Translation Validity • Face Validity: plausible on its “face” • Content Validity: matches lists of attributes • Criterion-related Validity • Predictive Validity: predicts accurately • Concurrent Validity: distinguishes appropriately between groups • Convergent Validity • Discriminant Validity

27. The Convergent Principle Alternative measures of a construct should be strongly correlated.

28. Self-esteem construct Item 1 Item 2 Item 3 Item 4 How It Works Theory You theorize that the items all reflect self-esteem.

29. Self-esteem construct Item 1 Item 2 Item 3 Item 4 How It Works Theory 1.00 .83 .89 .91 .83 1.00 .85 .90 .89 .85 1.00 .86 .91 .90 .86 1.00 The correlations provide evidence that the items all converge on the same construct. Observation

30. Convergent Validity in Measures of “Democracy” 1985| polity2 pollib civlib reg -------------+------------------------------------ polity2 | 1.0000 -0.9148 -0.8770 -0.8601 pollib | -0.9148 1.0000 0.9176 0.8440 civlib | -0.8770 0.9176 1.0000 0.8053 reg | -0.8601 0.8440 0.8053 1.0000

31. Convergent Validity in Measures of “Education” 1985| 1 2 3 4 5 6 -------------+------------------------------------------------------ Ed. spending | 1.0000 -0.1217 0.2415 0.3563 0.0214 0.0195 Illiteracy (%) | -0.1217 1.0000 -0.5797 -0.7306 -0.8569 -0.6196 Cohort to Grade 4 | 0.2415 -0.5797 1.0000 0.4419 0.6553 0.3654 % Grade School | 0.3563 -0.7306 0.4419 1.0000 0.6230 0.3612 % Secondary School | 0.0214 -0.8569 0.6553 0.6230 1.0000 0.7576 % College | 0.0195 -0.6196 0.3654 0.3612 0.7576 1.0000

32. The Discriminant Principle Measures of different constructs should not correlate highly with each other.

33. Self-esteem construct SE1 SE2 How It Works Theory Locus-of-control construct LOC1 LOC2

34. Self- esteem construct SE1 SE2 You theorize that you have two distinguishable constructs. How It Works Theory Locus-of-control construct LOC1 LOC2

35. Self-esteem construct SE1 SE2 How It Works Theory Locus-of-control construct LOC1 LOC2 rSE1, LOC1 = .12 The correlations provide evidence that the items on the two tests discriminate. rSE1, LOC2 = .09 rSE2, LOC1 = .04 rSE2, LOC2 = .11 Observation

36. We have two constructs. We want to measure self-esteem and locus of control. Theory Self-esteem construct Locus-of-control construct SE1 SE2 SE3 LOC1 LOC2 LOC3 For each construct, we develop three scale items; our theory is that items within the construct will converge and Items across constructs will discriminate.

37. SE1 SE2 SE3 LOC1 LOC2 LOC3 1.00 .83 .89 .02 .12 .09 .83 1.00 .85 .05 .11 .03 .89 .85 1.00 .04 .00 .06 .02 .05 .04 1.00 .84 .93 .12 .11 .00 .84 1.00 .91 .09 .03 .06 .93 .91 1.00 SE1 SE2 SE3 LOC1 LOC2 LOC3 Theory Locus-of-control construct Self-esteem Construct Green and red correlations are Convergent; yellow are Discriminant. SE1 SE2 SE3 LOC1 LOC2 LOC3 Observation

38. SE1 SE2 SE3 LOC1 LOC2 LOC3 1.00 .83 .89 .02 .12 .09 .83 1.00 .85 .05 .11 .03 .89 .85 1.00 .04 .00 .06 .02 .05 .04 1.00 .84 .93 .12 .11 .00 .84 1.00 .91 .09 .03 .06 .93 .91 1.00 SE1 SE2 SE3 LOC1 LOC2 LOC3 Theory Self-esteem construct Locus-of-control construct SE1 SE2 SE3 LOC1 LOC2 LOC3 The correlations support both convergence and discrimination, and therefore construct validity. Observation

39. What Is Reliability? • The “repeatability” of a measure • The “consistency” of a measure • The “dependability” of a measure

40. Rating Sheet 1 Manage time effectively 2 Manage resources effectively. 3 Scan a multitude of information and decide what is important. 1 2 3 4 5 4 Decide how to manage multiple tasks. 5 Organize the work when directions are not specific. 1 2 3 4 5 1 2 3 4 5 3 Scan a multitude of information and decide what is important. 1 Manage time effectively 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 True Score Theory Observed score True ability Random error + = X e + T

41. The Error Component X e + T Two components: er • Random error es • Systematic error

42. The Revised True Score Model X er es + + T

43. Random Error Frequency The distribution of X with no random error X

44. Random Error The distribution of X with random error Frequency The distribution of X with no random error Notice that random error doesn’t affect the average, only the variability around the average. X

45. Systematic Error Frequency The distribution of X with no systematic error X

46. Systematic Error The distribution of X with systematic error Frequency The distribution of X with no systematic error Notice that systematic error does affect the average; we call this a bias. X

47. If a Measure Is Reliable... We should see that a person’s score on the same test given twice is similar (assuming the trait being measured isn’t changing). X1 X2

48. If a Measure Is Reliable... But, if the scores are similar, why are they similar? X1 X2 Recall from true score theory that... T + e1 T + e2

49. If a Measure Is Reliable... The only thing common to the two measures is the true score, T. Therefore, the true score must determine the reliability. X1 X2 T + e1 T + e2

50. Reliability Is... a ratio variance of the true scores variance of the measure var(T) var(X)