Elizabeth A. Witt, Ph.D. American Board of Emergency Medicine

1. Expect the Unexpected: Are We Clearly Prepared? Effective Communication of Exam Results: What Should (or Shouldn't) be Included in the Candidate's Score Report Elizabeth A. Witt, Ph.D. American Board of Emergency Medicine Council on Licensure, Enforcement and Regulation 2006 Annual Conference Alexandria, Virginia

2. Validity • Degree to which evidence supports the interpretation and proposed use of test scores Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

3. Certified Chefs of America Examination Results Exam: South American Cuisine Date: August 15, 2006 Name: Jane Smith Result: Pass Your certificate will be mailed within 10 days. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

4. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

5. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

6. What information do examinees want? • Pass/Fail status • Actual score78 • Comparison to passing score • Areas of strength/weakness • Comparison with others Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

7. What else do examinees want? Timely notification Simplicity Explanations – graphs, technical terminology Graphics, visual presentation Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

8. Why do we not just give them what they want? Timely notification Instant notification possible with CBT BUT Not with paper exams AND Requires relinquishing control INCLUDING Elimination of poorly performing items Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

9. Why do we not just give them what they want? Simplicity We lose sight of the purpose/audience. This is not a technical report! Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

10. Why do we not just give them what they want? Explanations– graphs, technical terminology Not always necessary Familiarity – lose sight of the audience May require psychometric services Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

11. Why do we not just give them what they want? Graphics, visual presentation Not always necessary Small staff, small program Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

12. Why do we not just give them what they want? Actual score – may need explanation Raw score: number correct, percent correct Logits Scaled score Scores may not be precise away from the cut Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

13. Why do we not just give them what they want? • Comparison to passing score • –Scores may not be precise except at the cut • –May need explanation • (SEM, for example)– May be self-explanatory • (if a simple raw is reported, for example) Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

14. Why do we not just give them what they want? Areas of strength/weakness - Need a subscore for each area - Scores based on few items are not reliable - Subscores may not sum to total test score - confusing to examinees Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

15. Why do we not just give them what they want? Comparison with others - Scores are normally criterion-referenced. - The group testing may differ by exam administration. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

16. Recommendations DO include: • Name, exam title, exam date • Pass/Fail Status • An interpretable score • Score required to pass • Same results by section, if appropriate • Simple, clean graphics Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

17. Recommendations DO include: • Simple, clear explanations for any graphs or technical terminology • Brief, simple explanation of scoring • Diagnostic information for those who fail • How to request score verification, appeal result Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

18. Recommendations Do NOT include: • Decimals • Mean score = 275.358 • Mean score = 275 • Excessive detail • Jargon • SEMCoefficient AlphaLogit • Personal information • (SSN, for example) • Teeny, tiny font sizes Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

19. Recommendations Candidate score reports should be: • Timely • Simple • Clear • Brief • Easy to read and understand Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

20. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

21. Bottom Line Who is the audience and what do they need to know? Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

22. References/Resources • American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association • Ryan, J. M. (2006). Practices, issues, and trends in student test score reporting. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of Test Development (pp. 677-710). Mahwah, NJ: Lawrence Erlbaum. • Schroeder, L. L. & Castle, R. Technology and testing: Reporting scores. (Winter 2006). CLEAR Exam Review, 27(1), 8-10. • http://www.ets.org/Media/Products/IDMS/pdf/51588_idmsRpt.pdf • http://www.realestatesimulator.com/pages/sample_reports/Sample3.htm • http://www.visualcertexam.com/screenshots.html Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

23. Speaker Contact Information Elizabeth A. Witt, Ph.D. American Board of Emergency Medicine 3000 Coolidge Rd. East Lansing, MI 48823 517-332-4800, ext. 328 ewitt@ABEM.org www.ABEM.org Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia