Site Visit 102: The DIO’s Role for Program Site Visits Ann M. Dohn, M.A. DIO Stanford
Disclaimer • No Conflicts of Interest to Report
DIO Competencies • Maintaining the Institution’s Residency Program’s ACGME accreditation • Improving the Institution’s education program • Developing and supporting residency program directors
DIO Vision • “Your job is not a powerful one…your job is to empower others.” i.e. program directors • Helen Rantz, 1984
Session Objectives • To understand the role of the DIO in ACGME program site visits • Using ACGME annual resident and faculty surveys • Using internal reviews and annual program evaluation in preparation for site visits • Using other potential measures of program quality
Session Objectives • A time of change… • Living in two worlds-starting 7/1/2013 • “PIF-less Site visits / NAS • Site visits with PIFs ACGME OLD NEW
Then and Now • 1984 site visits vs. 2012 site visits
Start with the New and “unknown” • Unknown is scary
New Model: What Do We Know? • Longer Cycles • Annual WebADS Updates • ACGME Resident and Faculty Surveys • End of the PIF
End of the PIF • DIOs will have more recreational reading time!
“New” Model • Effective 7/1/2013 • “Early adaptors”= RRCs deciding to go “PIF-less” • Emergency Medicine • Internal Medicine • Neurosurgery • Orthopedic Surgery • Pediatrics • Radiology (Diagnostic) • Urology • and their subs
New Accreditation System (NAS) • The NAS requires categorization of the ACGME’s Common Program Requirements to clearly identify: • Core Requirements • Detail Requirements • Outcome Requirements
New Accreditation System (NAS) • Core Requirements • Statements that define structure, resource, or process elements essential to every graduate medical educational program. • Duty Hours • Sponsoring Institution’s responsibility for Program Accreditation • Change in program director • Qualifications of the program director
New Accreditation System (NAS) • Detail Requirements: • Statements that describe a specific structure, resource, or process for achieving compliance with a Core Requirement.
Examples of “Detail” Requirements • Detail Requirements include particular educational approaches and learning experiences • a given number of lectures or hours of bedside teaching experiences • Ensuring compliance with grievance and due process procedures as set forth in the Institutional Requirements and implemented by the sponsoring institution
New Accreditation System (NAS) • Outcome Requirements: • Statements that specify expected measurable or observable attributes (knowledge, abilities, skills, or attitudes) of residents or fellows at key stages of their graduate medical education. • Programs in substantial compliance with the Outcome Requirements may utilize alternative or innovative approaches to meet Core Requirements.
Example of “Outcome” Requirements • Outcome requirements include: • Milestones • Residents must be able to competently perform all medical, diagnostic, and surgical procedures considered essential for the area of practice.
New Model-Tracer Model-Inverted Model • Increases value on resident input • ACGME survey • Resident interviews • Focuses on PROGRAM rather than a description of the program • The Tolstoy novel will no longer be written!!
New Model-PIF Questions • Small number of questions from PIF now in annual WebADS update • Remaining information assessed by the field staff
Tracer Method • Allows for the selection of particular elements to be evaluated: • Citations • Complaints from residents/faculty • Non-compliance reported on the ACGME resident survey • Duty hour and learning environment standards
Tracer Method • Focuses on: • Program’s actual operations • Implementation of policies and procedures
Tracer Method – Interactive – A Key Change • Allows for discussion of important topics related to the new duty hour standards • Provides opportunities for site visitors to educate program leaders and share innovative practices
Tracer Method • Longitudinal information on: • New standards • Prior citations/corrective actions • Resident survey identified areas of non-compliance • Areas identified during preparation for the Site Visit
Tracer Method • Will focus on relevant high-priority issues for the given program • Planned for two-four areas/topics per program • Done during the regular interview (no walkarounds)
Tracer Method • Enhancing resident input • Residents to directly respond to the site visitor with a single, confidential “consensus list” • strengths • areas for improvements • starts with the trainees
PIF-Less Site Visit: The Stanford Experience • August 2011 • Three Programs • Visit done in lieu of mid-cycle internal review • Data not used for accreditation • Focus was on program requirements • Used comprehensive data system (no paper)
PIF-Less Site Visit: The Stanford Experience (2) • Individuals interviewed = regular site visit • ACGME provided comprehensive reports • Extensive debriefing with the site visitors and program participants • Very positive experience for PDs. PCs, and the DIO and GME staff
PIF-Less Site Visit: Stanford Findings • PIF-Less (inverted) site visit report finding • Comparable to standard internal review • Note: our protocol always starts with the residents/fellows
PIF-Less Site Visit: ACGME Findings • Findings: • Residents feel more engaged • PDs love it • Early ID of Program Themes • Improves continuity and integration of data • Seamless process for follow-up • Increased opportunity for reconciliation of discrepant issues • Transformative Model - rather than additive
PIF-Less Site Visit: ACGME Findings (2) Findings Continued • Comparable to visit with PIF for citation review • Superior for review of program improvements and positive program elements • Variability with other elements • Dependent on data system used by the institution/program
Upcoming Changes in the Site Visit • Broader implementation of the inverted Site Visit • Team site visits (two site visitors) • Programs • Institutions
ACGME Use of Survey Data during the Site Visit • Use resident data to validate/clarify program information • Compare resident data to program director and faculty interviews • Validate resident data to program documentation • Compare duty hours responses to institutional data
DIO Use of Data for Program Site Visits • ACGME resident and faculty surveys • APRs • Internal Reviews • Other measures of program quality
ACGME Concerns with the Resident Survey • ACGME concerns with the resident survey: • “Honesty” in reporting • ? Residents being coached ? • Amount of time spent by residents completing the survey = 20 seconds per question • Survey questions being sold to companies who want to ‘help’ programs prepare • Survey fatigue
ACGME Annual Resident Survey • Possible solutions to using the survey • 2012- Aggregation of individual survey questions into groupings • 2012-use of alternate forms of the survey • Educating residents about the vital nature of the survey
Other Possible Sources of Data • GME Annual House Staff Survey • Duty Hours Reports • Evaluations of programs, faculty, and house staff • Alumni surveys • Board and in-training exam scores
GME Annual House Staff Survey • Provides feedback on eight (8) key areas of Graduate Medical Education: • Overall Experience • Program Curriculum • Program Faculty • Evaluation and Feedback • Training Environment • Personal Wellness • Quality Improvements • Communication and Patient Perceptions • Other areas….(anything we missed)
GME Annual House Staff Survey • Provides a level of detail not available from the ACGME survey • Such as on call room issues, fear/intimidation issues, harassment issues, faculty • Allows for text answers, comments Warning: Don’t do a survey unless you are willing to accept and act on the input