1. Reliability
2. Todays Objectives Define measurement error and describe how it interferes with psychological studies
Know that reliability is the ratio of true variability to observed variability
Name major sources of error variance
Describe methods for assessing reliability
3. Reliability In psychometrics, reliability refers to consistency in measurement
4. Error Variance Score X is a reflection of the true score on the psychological construct being measured plus error.
X = T + E
5. Error Variance If we wanted to measure spelling ability, the best technique would be to systematically go through the dictionary and have a person spell every word, which would be the TRUE score for spelling ability.
But, that would be a terrible way to spend the summer.
Better method would be to take a random sample of words from the dictionary.
Reliability analysis is estimating how much error we would make by using the score from the shorter test as an estimate of spelling ability.
6. Error Variance s 2 = s T 2 + s E 2 + s S 2 The greater the proportion of variance attributed to the s E , the less reliable the test.
7. Error Variance
8. Error Variance
This item is probably an unreliable measure of a persons feelings toward iguanas.
Why? Ample individual differences abound regarding feelings toward mating.
9. Measuring Reliability Suppose you measured the height of every member of your family with a crude stick.
The goal is to reliably distinguish between individuals.
What would be more reliable: Measuring everyone once, or measuring each person 100 times and then taking the average across the 100 measurements as the final measure of each persons height?
10. Measuring Reliability Principle Numero Uno:
The more items on a test, the better the reliability.
This is because errors will be randomly distributed across items, and sum to zero.
For example, when given Flying iguanas are ugly as a test item, people who are for/against mating will be randomly for/against flying.
11. Measuring Reliability
12. Measuring Reliability
13. Sources of Error Variance Test construction
Test administration
Test scoring and interpretation
14. Sources of Error Variance Misinterpretation
Social Desirability
Misunderstanding the Instructions
Attention Seeking
Forgetting
15. Reliability Estimates Test-Retest Method
The test is administered twice at two different points in time.
Assumes that there will be no change in the quality or construct being measured.
Test-retest reliability is best used for things that are stable over time, such as intelligence.
16. Reliability Estimates Parallel-Forms/Alternate Forms Method
Parallel tests have identical true scores and identical error variances.
Alternate forms are two forms of a tests whose items are intended to measure the same thing and do not differ from each other in any systematic way.
E.g., Watson-Glaser Critical thinking test items:
Test A: Terry don't worry about it. You'll graduate someday. You're a college student. Right? And all college students graduate sooner or later.
Test B: Charlie, don't worry about it. You'll get a promotion someday. You're working for a good company. Right?. And everyone who works for a good company gets a promotion sooner or later.
17. Reliability Estimates Split Half
Correlate scores from equivalent halves of a single test administered once.
Goal is usually to create parallel forms.
Divide the items on the test into equivalent halves
Correlate the scores between the two halves
Adjust the result to take into account that if you were to reduce the items, reliability would decrease (Spearman-Brown formula)
18. Reliability Estimates Interrater Reliability Method
Evaluate the consistency among judges who are rating the same behavior
19. Summary
