Redefining Simulator Proficiency: Assessing Expertise with Automaticity Theory in Laparoscopic Training

1. Redefining Simulator Proficiency Using Automaticity Theory Dimitris Stefanidis, MD, PhD, Mark W Scerbo, PhD James R Korndorffer Jr, MD, Daniel J Scott, MD Carolinas Medical Center, Charlotte, NC Old Dominion University, Norfolk, VA Tulane University, New Orleans, LA UT Southwestern Medical Center, Dallas, TX

2. Background • Training to proficiency on simulators improves operative performance1,2 • Proficiency levels are derived from expert performance • Most common performance metrics used are time and errors • Expertise is, however, not well defined in surgery3 and other metrics may more accurately assess performance 1 Korndorffer Jr JR, JACS 2005 2 Seymour NE, Ann Surg 2002 3 Ericsson KA, Acad Med 2004

3. Background • Automaticity refers to the habitual performance of a task without significant demands on attention1 • Expert performance is characterized by automaticity; it confers them the ability to multi-task • A secondary task performed simultaneously with the primary task can be used as an index of expertise by measuring multi-tasking ability 1 Schiffrin & Schneider, Psychol Rev 1977

4. Background • Secondary tasks must be sensitive, selective and unobtrusive1 • They must compete for resources that are common to the primary task • Laparoscopy imposes heavy visual-spatial demands on the surgeon • A visual-spatial secondary task that draws from the same attentional resources is ideal 1 O’Donnell & Eggemeier, 1986

5. Study Objective • To assess whether a visual-spatial task that measures attentional spare capacity can distinguish among subjects of variable expertise in laparoscopic suturing

6. Methods • IRB- approved protocol • 12 Participants: 4 novices, 3 surgery residents, 3 laparoscopy experts, and 2 novices trained to proficiency in laparoscopic suturing • Participants had to perform under dual-task conditions for 10 minutes • Laparoscopic suturing and knot tying on a videotrainer simulator using the FLS model • Visual-spatial secondary task performance

7. Video - Primary task Expert Novice

8. Video - Secondary task

9. Video - Dual Tasking

10. Metrics • Laparoscopic Suturing • Objective score (=300-[time+10*errors]) • Time (max 5 min per repetition) • Errors • Accuracy • Knot Security • Secondary task • Percent correct detections • When performed alone • When performed simultaneously with suturing

11. Statistical Analysis • Kruskal-Wallis One Way Analysis of Variance on Ranks for the four groups with pairwise comparisons (Dunn’s method)

12. Results • Experts:>200 basic and >100 advanced laparoscopic cases, extensive experience with simulator • Novices: no prior laparoscopy or simulator experience • Trained: achieved proficiency on simulator, no operative experience • Residents: <100 basic, <30 advanced laparoscopic cases, familiar with simulator

13. Results • All achieved a perfect score on secondary task when performed alone (100% correct detections) • Secondary task performance deteriorated in all groups when performed simultaneously with primary task (p<0.001) • Experts and trained outperformed residents and novices on suturing task • While experts did not differ from trained based on time and errors, they achieved better secondary task scores (p=n.s.)

17. Conclusions • A visual-spatial secondary task that assesses attentional spare capacity may distinguish among levels of laparoscopic expertise (construct validity) when standard performance measures fail to do so • Such secondary task metrics may more accurately define expert performance for use as training endpoints during simulator curricula and possibly for assessment purposes • Further validation of secondary task metrics and automaticity is warranted and currently underway

18. ?