1. Web-Enabled Evaluation Techniques: Applying a Competency-Based Curriculum Charles O. Frazier, M.D. Jon J. Kaminer, M.D. AFPRD Workshop

2. Objectives • Why effective evaluation is critical • How the ACGME guidelines can fit in an effective evaluation scheme • How a web-enabled evaluation tool developed at Riverside can be used to apply a competency-based curriculum AFPRD Workshop

3. Why is an evaluation system needed? • It’s a requirement • Further defines expectations • Promotes goal directed behavior • Provides comfortable atmosphere for constructive criticism and positive reinforcement AFPRD Workshop

4. What is effective evaluation? • Specific/focused • Timely • Consistent • Target behavior/performance reasonable • Conducted in collegial manner • Ends with defined goal/expectation of future behavior/performance AFPRD Workshop

5. What are the main evaluation modalities? • feedback • summative evaluation AFPRD Workshop

6. Feedback • Immediate comment AFPRD Workshop

7. Summative Evaluation • Longitudinal measure of performance AFPRD Workshop

8. Modality Comparison AFPRD Workshop

9. How does the ACGME outcomes project fit in to this evaluation framework? AFPRD Workshop

10. Competencies AFPRD Workshop

11. List of ACGME competencies • Patient Care • Medical Knowledge • Practice-Based Learning and Improvement • Interpersonal and Communication Skills • Professionalism • Systems-Based Practice AFPRD Workshop

12. www.ACGME.org/Outcome AFPRD Workshop

13. Toolbox • Record review • Chart Stimulated recall • Checklist • Global rating • SP • OSCE • Simulations and Models AFPRD Workshop

14. Toolbox • 360º Global Rating • Portfolios • Exam MCQ • Exam oral • Procedure case or logs • Patient survey AFPRD Workshop

15. Benefits • defines the ACGME’s expectations of us • Defines our expectations for residents to graduate • allows for consistency • changes focus of evaluation to resident performance rather than residency structure AFPRD Workshop

16. Limitations • Incredible amount of work and paper work • How will we use all this information to provide meaningful feedback to the resident • Will this take away from resident teaching time AFPRD Workshop

17. Riverside Family Practice Residency AFPRD Workshop

18. Our former evaluation framework Feedback Precepting AFPRD Workshop

19. Our former evaluation framework Summative Evaluation • Trimester Faculty meetings • Trimester directors meetings • Rotational evals • Full faculty impression AFPRD Workshop

20. Pros • Plenty of face-time and feedback during precepting • Gestalt of faculty about resident shared and discussed • Not much paperwork for faculty AFPRD Workshop

21. Cons • No specifics • Summative evaluation disconnected from feedback • Inefficient • No good individualized action plan • Overloaded residency director AFPRD Workshop

22. Structural Changes Made Advising - March 2001 • Advisor system • Quarterly advisor meetings with Educational Prescription • Semiannual director meetings AFPRD Workshop

23. Competency-based System • Paper evaluation forms developed in light of competencies • Expectations defined for residents and faculty • Our residency “toolbox” was expanded and redefined based on ACGME guidelines AFPRD Workshop

24. Page 425 Tool Matrix

25. Precepting DOP Rotations Nurses Patients Procedure Report Record Completion Report Peers Research Project Care Managers Medical Students

26. Precepting DOP Rotations Nurses Patients Procedure Report Online Comp-Eval Server Record Completion Report Peers Research Project Care Managers Medical Students Faculty Resident

27. August/September - Nurse Evaluation / Faculty Advisor Evaluation November/December - Nurse Evaluation / Faculty Advisor Evaluation December - Semi-Annual Program Director Evaluation February/March - Nurse Evaluation / Faculty Advisor Evaluation May/June - Nurse Evaluation / Faculty Advisor Evaluation June - Semi-Annual Program Director Evaluation AFPRD Workshop

28. Web-Enabled System • A stand-alone evaluation system • Universally accessible • Highly customizable • Data automatically collected in database • stored, studied, and shared AFPRD Workshop

29. Technical description - Intranet • Compaq Proliant Server • Microsoft Windows NT 4.0 • Microsoft Internet Information Services 5.0 • Microsoft FrontPage 2000 • Active Server Pages • Microsoft Internet Explorer AFPRD Workshop

30. Active Server Pages • Script (programming) is executed on the server, and resulting HTML is sent back to and displayed on browser • Connects to database • Underlying script secure AFPRD Workshop

31. Web Browser Database AFPRD Workshop

32. Technical Description - Intranet • Security • NT Server Authentication • Cost • Server $3,000 • Software $1500 • Labor (estimate) 450 hours AFPRD Workshop

33. Technical Description - Internet • VCU IIS Public Information Server • Security with ASP database passwords • No direct cost to residency • Domain name registration • www.pubinfo.vcu.edu/webeval AFPRD Workshop

34. Demonstration AFPRD Workshop

35. The future • Ability to study competency-based education • Evaluate Program Curriculum • Ability to describe expected “Natural History” or “Growth Curve” for resident • Generate documentation to support putting a resident on academic probation AFPRD Workshop

36. Questions? http://www.pubinfo.vcu.edu/webeval charles.frazier@rivhs.com jon.kaminer@rivhs.com AFPRD Workshop