“Problems and Priorities in Pretesting Pondered”

1. “Problems and Priorities in Pretesting Pondered” Beth Noeller Grady Barnhill Carol O’Byrne

2. CLEAR 2008 Annual Conference Anchorage, Alaska

3. All Items are not Created Equal CLEAR 2008 Annual Conference Anchorage, Alaska

4. Purposes of Pretesting • Gather performance information • Add new content CLEAR 2008 Annual Conference Anchorage, Alaska

5. What Information do you Gather? • Performance data • Classical and IRT • Difficulty • Discrimination • Cognitive Level Data • Content CLEAR 2008 Annual Conference Anchorage, Alaska

6. Difficulty • How hard is the item? • Usually expressed as a %. • What percent of the candidates get the item correct? • IRT b-value CLEAR 2008 Annual Conference Anchorage, Alaska

7. What is a “good” difficulty? • What is your cut score? • The best difficulty is right around the cut score. • Too easy and too hard provide little information. • Between 40 & 90 for many testing programs CLEAR 2008 Annual Conference Anchorage, Alaska

8. Discrimination • How well an item differentiates between those who perform well on the test and those who get the item correct. • Correlation • Values from -1.0 to +1.0 • IRT a-value CLEAR 2008 Annual Conference Anchorage, Alaska

9. Positive Discrimination Values • Two people walking together ‘in step’ • Get an item correct and receive a high score on the examination • Perfect positive relationship +1.0 exam score Item correctness CLEAR 2008 Annual Conference Anchorage, Alaska

10. Negative Discrimination Values • Two people walking in opposite directions • Get an item correct and receive a low score on the examination • Perfect negative relationship -1.0 Item correctness CLEAR 2008 Annual Conference Anchorage, Alaska exam score

11. What is a “good” discrimination? • A positive relationship • The higher the better • Diversity of the candidate population • Greater than 0.15 Item correctness exam score CLEAR 2008 Annual Conference Anchorage, Alaska

12. Combining Difficulty and Disc. • Each quality by itself is linear • Together they can be depicted on a two-dimensional graph CLEAR 2008 Annual Conference Anchorage, Alaska

13. CLEAR 2008 Annual Conference Anchorage, Alaska

14. CLEAR 2008 Annual Conference Anchorage, Alaska

15. CLEAR 2008 Annual Conference Anchorage, Alaska

16. CLEAR 2008 Annual Conference Anchorage, Alaska

17. CLEAR 2008 Annual Conference Anchorage, Alaska

18. CLEAR 2008 Annual Conference Anchorage, Alaska

19. CLEAR 2008 Annual Conference Anchorage, Alaska

20. CLEAR 2008 Annual Conference Anchorage, Alaska

21. CLEAR 2008 Annual Conference Anchorage, Alaska

22. CLEAR 2008 Annual Conference Anchorage, Alaska

23. CLEAR 2008 Annual Conference Anchorage, Alaska

24. CLEAR 2008 Annual Conference Anchorage, Alaska

25. CLEAR 2008 Annual Conference Anchorage, Alaska

26. CLEAR 2008 Annual Conference Anchorage, Alaska

27. CLEAR 2008 Annual Conference Anchorage, Alaska

28. CLEAR 2008 Annual Conference Anchorage, Alaska

29. CLEAR 2008 Annual Conference Anchorage, Alaska

30. Summary CLEAR 2008 Annual Conference Anchorage, Alaska

31. How do you use pretesting to add new content? • Get the pulse of your item bank • Inventory of acceptable scored items • Inventory of unused items CLEAR 2008 Annual Conference Anchorage, Alaska

32. Scored Items and Test Specs CLEAR 2008 Annual Conference Anchorage, Alaska

33. Pretest Needs CLEAR 2008 Annual Conference Anchorage, Alaska

34. Beth’s Contact Information Beth Noeller, PhD 319-339-3095 beth.noeller@act.org CLEAR 2008 Annual Conference Anchorage, Alaska