The ABC’s of Pattern Scoring

1. The ABC’s of Pattern Scoring Dr. Cornelia Orr

2. Vocabulary • Measurement – Psychometrics is a type of measurement • Classical test theory • Item Response Theory – IRT (AKA logistic trait theory) • 1, 2, & 3-parameter IRT models • Pattern Scoring

3. Measurement Assign numbers to objects or events Ex. – hurricanes, earthquakes, time, stock market, height, weight Psychometrics Assigning numbers to psychological characteristics Ex. – achievement personality, IQ, opinion, interests General & Specialized

4. Classical Test Theory Item discrimination values Item difficulty values (p-values) Guessing (penalty) Number correct scoring Item Response Theory Item discrimination values Item difficulty values Guessing (pseudo-guessing) values Pattern scoring Different Theories of Psychometrics Similar constructs – Different derivations

5. Number-Correct Scoring Simple Mathematics Raw scores (# of points) Mean, SD, SEM, % correct Number right scale Score conversions Scale scores, percentile ranks, etc. Pattern Scoring Complex Mathematics Maximum likelihood estimates Item statistics, student’s answer pattern, SEM Theta scale (mean=0, standard dev=1) Score conversions Scale scores, percentile ranks, etc. Different Methodsof Scoring

6. Similarities The relationship of derived scores is the same, e.g., High correlation, (0.95) of number right scores and scale scores Scale score has the same percentile rank for both methods Differences Methods of deriving scores The number of scale scores possible Number right = limited to the number of items IRT = unlimited or is limited by the scale (ex. 100-500) Comparison: Number Correct and Pattern Scoring

7. Choosing the Scoring Method • Which model? • Simple vs. Complex? • Best estimates? • Advantages/Disadvantages? Ex. – Why do the same number correct get different scale scores? Ex. – Flat screen TV – how do they do that?

8. Advantages of IRT and Pattern Scoring • Better estimates of an examinee’s ability • the score that is most likely, given the student’s responses to the questions on the test (maximum likelihood scoring) • More information about students and items are used • More reliability than number right scoring • Less measurement error (SEM)

9. Disadvantages of IRT and Pattern Scoring • Technical - Complex Mathematics – • Difficult to understand • Difficult to explain • Not common – Not like my experience. • Perceived as “Hocus Pocus”

10. Item Characteristic Curve (ICC)

11. Effect Of Item Difficulty No Typea b c 1 1 MC 0.0150 250.000 0.1 2 1 MC 0.0150 275.000 0.1 3 1 MC 0.0150 300.000 0.1 4 1 MC 0.0150 325.000 0.1 5 1 MC 0.0150 350.000 0.1 Response Patterns (1=correct) Pattern SEM SS 12345 11100 43 300 01110 43 305 10101 43 305 43 310 Answering more difficult items (b-parameter) can result in higher scores. Examples

12. 5 Items (Effects of Item Discrimination) No Typea b c 1 MC 0.0050 300.000 0.2 2 MC 0.0100 300.000 0.2 3 MC 0.0150 300.000 0.2 4 MC 0.0200 300.000 0.2 5 MC 0.0250 300.000 0.2 Response patterns (1=correct) Pattern SEM SS 12345 11001 94 280 11100 61 270 01110 46 300 00111 39 305 Answering more discriminating items (a-parameter) can result in higher scores. Examples