Michael Corneille MD, Ross Willis PhD, Ronald Stewart MD, Daniel Dent MD

1. Performance on Brief Practice Exam Identifies residents at Risk for Poor ABSITE and ABS Qualifying Exam Performance Michael Corneille MD, Ross Willis PhD, Ronald Stewart MD, Daniel Dent MD

2. Background • ABSITE • First offered 1975 to gauge residents’ acquisition of fundamental surgical knowledge • Not designed as predictive tool • Strong correlation between performances on ABSITE and ABS QE • Shellito et al. (2010) Am J Surg • No pass/fail cutoff • ABSITE vs ABS QE pass rates • ABSITE >30th %ile - 93% 1st attempt QE passed • ABSITE ≤30th %ile - 73% 1st attempt QE passed • 2009 ABSITE • .51 correlation between score and QE score (1st time) • .92 correlation between score and QE pass (1st time)

3. Purpose of Practice Exams • Act as a study tool • Identify residents at risk for poor performance on ABSITE and QE • 2 part exercise • Constructing the exam • Taking the exam • Objectives based on potential reasons for poor performance

5. Background • Practice test not designed as a predictive tool • Same guidelines as ABSITE • Scores not used to determine promotion • Allow program director to objectively compare resident’s fund of knowledge to peers

6. Hypothesis • We lacked objective data to counsel residents as to true interpretation of practice test scores • Hypothesis • A brief practice exam • May identify residents at risk of poor ABSITE performance • May identify early in the chief resident year those at risk for poor QE performance.

7. Methods • Study period 2005-2010 • RRC approved to graduate 9 chiefs/year • 1-3 practice exams/year • 60 mins/exam (50-55 basic science & clinical questions) • Questions based on the ABSITE keywords authored by residents and edited by faculty • Assigned 3 to 5 topics • Standardized format • Question • 5 response options • Explanations with references • Optically graded

8. Methods • Exam was considered mandatory • Educational time allotted within the training program • Scores reported to residents within two weeks • % correct and standard deviation within PGY class • Residents >1 standard deviation below the mean for their class counseled • Expectation of the program > 30th %ile reiterated • Referred to a faculty mentor to develop study plan

9. Data Analysis • Compared resident’s practice exam most recently preceding ABSITE to that year’s ABSITE • Compared chief year ABSITE to ABS QE Pass/fail status • Spearman rank correlation coefficients used to compare dependent measures

10. Results • 9 practice exams administered to all resident levels • 432 individual practice exams completed • 309 residents have taken the ABSITE • 33 have taken the ABS QE • 263 residents completed at least one practice exam in the year preceding the ABSITE • 26 chief residents completed at least one practice exam immediately preceding the ABS QE

11. Practice Test vs ABSITE

12. Data Analysis • Correlation between Practice Exam and first attempt QE score was significant • (r = .416, p < .05) • Correlation between chief year ABSITE and the first attempt QE score was significant • (r = .743, p < .01) • ABSITE score <30th %ile • 89% passed the QE on the first attempt

13. Discussion • What to do with the data? • Can we intervene? • Is the exercise beneficial?

14. Discussion • Several authors have reported that intervention can improve scores 1,2 • Our data show that performance on practice exam is correlated to ABSITE and ABS QE • May suggest that our interventions are inadequate or that there is a cohort of residents destined to struggle with standardized assessments 1Borman KR, Does Academic Intervention Impact ABS Qualifying Examination Results? Current Surgery, Vol 63, Issue 6, Pp 367-372, Nov 2006 2Hirvela ER, Becker DR Impact of Programmed Reading on ABSITE Performance. The American Journal Of Surgery, Vol 162 Nov 1991)

15. Discussion • Silver lining • May seem overall pessimistic • Those who do struggle may be excellent surgeons • Those who struggle may pass their boards on the first attempt • 89% first attempt pass with ABSITE scores less than 30th percentile

16. Conclusions • Poor performance on a brief practice exam • Does identify residents at risk for poor ABSITE performance • Does identify residents at risk for poor ABS QE performance and earlier than ABSITE scores.

17. Conclusions • More data is necessary on what interventions truly work to improve ABS QE pass rates in very high risk residents

18. Thank You