Measurement Theory & Construct Validity

1. Measurement Theory & Construct Validity Chapter 3

2. Measurement validity = construct validity

3. Construct Validity

4. Construct Validity • Determined by Operationalization

5. Construct Validity • So construct validity assesses how well your procedures/measures match your ideas/theories • General = construct • Specific = operationalization

6. Construct Validity • Two views – • Definitionalist • “The construct,the whole construct, and nothing but the construct” • Impossible! • Relativist • Define your construct • Explain how and why what you’re doing measures the construct in question • Produce some evidence (whether we do this depends on the kind of research in question) US

7. Construct Validity • Translation validity • Face validity • Content validity • Criterion-related validity • Predictive validity • Concurrent validity • Convergent validity • Discriminant validity

8. Construct Validity • Translation validity vs. Criterion-related validity • Translation validity assesses whether the operationalization matches what you know of the construct • Criterion validity actually measures this assessment (uses other measures to assess the construct validity) What we do is translation validity – arguing about the construct validity but not measuring it directly

9. Construct Validity • Translation validity • Face validity: • Does it look like you got it right? • Ask others…more objective • Content validity • Good definition of the construct • Good match between your measure and the definition • E.G. Fitness program – does it abide by ACSM guidelines? (e.g. construct…Body fat. Measure…sum of skinfolds)

10. Construct Validity • Criterion-related validity – all involve some direct test of CV • Predictive validity • Does it predict what it ought to? • E.G. Does sum of skinfolds predict cardiovascular disease? • Concurrent validity • Can your measure discriminate between 2 similar groups? • E.G. Measure sum of skinfolds of males and females – females should be higher than males(?)

11. Construct Validity • Criterion-related validity • Convergent validity • Correlation between this operationalization and other similar ones • E.G. Sum of skinfolds and…BMI, underwater weighing, cadaver dissection… • Discriminant validity • This operationalization is different from other stuff that is not supposed to measure the same thing • E.G. Sum of skinfolds vs. age, vs. weight, vs. gender, and so on. • Note – this is why these items are included in popular equations converting sum of skinfolds to BF%

12. Construct validity • Convergent vs. Discriminant Validity

13. Construct validity • Convergent vs. Discriminant Validity • Note – once you’ve done this, you still need translation validity to establish that the measures are what you purport them to be

14. Threats to Construct Validity • The laundry list… • Use this to “ask the right questions” about the studies you critique • & so to begin…

15. Threats to Construct Validity • Inadequate preoperational explication of constructs • Construct not defined carefully enough

16. Threats to Construct Validity • Mono-operation bias (independent variable) • Only one example of the construct • E.G. only one training program…but there are many out there • Mono-method bias (dependent variable) • Only one example of the construct • E.G. only one strength measure for a program that trained the whole body

17. Threats to Construct Validity • Interaction of different treatments • Use one or more control groups to isolate cause • E.G. You want to show that strength training improves self-esteem…but it could have been as a result of meeting you 3 times per week, not strength training…so use a control group to compare results

18. Threats to Construct Validity • Interaction of testing and treatment • Imagine I was interested in whether research methods improved reasoning skills… • If I tested you every week on some IQ tests, these become part of the treatment, and impair construct validity (you might be getting better because of the test, not because of being in research methods)

19. Threats to Construct Validity • Restricted generalizability across constructs • “Unintended consequences” • E.G. Finnish epidemiological study – • divide people into groups according to level of smoking & cholesterol • Take half of each group and assign to fitness & nutrition program, half untreated • All in fitness and nutrition program reduced smoking and improved fitness measures • BUT they also had increased all cause mortality!!! (may be an apocryphal story)

20. Threats to Construct Validity • Confounding constructs and levels of constructs • “I’m finding out whether aerobics or strength training has the greater impact on muscle tone” • Ok, but what amount of strength training? How often are they doing aerobics? • The labels are not in sufficient detail • Could be that only the particular versions of these programs that you used will produce the results you found

21. Threats to Construct Validity • Social threats to construct validity • Hypothesis guessing • I never collect data using people that have completed my motor learning class… • Evaluation apprehension • Experimental booth in Bangor… • Experimenter expectancies • I love my research…

22. Reliability and Levels of Measurement • We’ll leave those till we deal with conclusion validity • They are really the concern of a good stats course