Framework for Excellence

2. Framework for Excellence Measuring Results Which helps in: Refining Site Analysis Marketing Curriculum Design Needs Assessment Course Delivery and Development Further Measurement and Evaluation!

3. Presenters Cheryl Hamill, RN, MS, ACRN & Nancy Showers, DSW Delta Region AETC HIVQual Results 2002-2003 Sample RW Title III Community Health Center in Mississippi Mari Millery, PhD NY/NJ AETC Lessons from Assessing Knowledge & Practice Outcomes of Level III Trainings Jennifer Gray, RN, PhD & Richard Vezina, MPH TX/OK AETC, Women & HIV Symposium (JG) Pacific AETC, Asilomar Faculty Development Conference (RV) Debbie Isenberg, MPH, CHES & Margaret Clawson, MPH Southeast AETC Intensive On-Site Training Evaluation: A Mixed Methods Approach Brad Boekeloo, PhD, ScM NMAETC, Delta AETC Analysis of HIV Patient-Provider Communication

4. Measurement and Evaluation Why evaluate? To determine if the training was successful in meeting aims (for participants and faculty) To decide how to change training content To improve the quality of training Why measure provider behavior change? To determine if training has the desired effect on participants and ultimately, on quality of care

5. Kirkpatrick�s Model (from Kirkpatrick, Donald L. Evaluating Training Programs (2nd edition) 1998)

6.  The HIVQUAL Project  Nancy Showers, DSW Delta Region AETC

7. The HIVQUAL Project Capacity�building and organizational support for QI Individualized on-site consultation services Strengthen HIV-specific QI structure Foster leadership support for quality Guide performance measurement Facilitate implementation of QI projects Train HIV staff in QI methods Performance measurement data with comparative reports Partnership with HRSA to support quality management in Ryan White CARE Act community-based programs

8. HIVQUAL Participants - 2003

9. Annual PAP Test

10. Annual Syphilis Screen

11. Hepatitis C Status Known

12. Adherence Discussed

13. Viral Load Every 4 Months 

14. MAC Prophylaxis (CD4<50)

15. Annual Dental Exam

16. Annual Mental Health Assessment

17. Delta AIDS Education and Training Center (DRAETC)Mississippi LPS - Training Summary ReportReporting period: July 1, 2002 - June 30, 2003for Targeted RW Title-Funded Community Health Centers Cheryl Hamill, MS, RN, ACRNInstructor of MedicineResource Center Directorhttp://hivcenter.library.umc.edu

19. Lessons from Assessing Knowledge and Practice Outcomes of Level III Trainings Mari Millery, PhD

20. Decided to focus more outcome evaluation efforts on Level III because it is the most intensive and a high priority modality; and participants can be asked to devote time to extra paperwork Pre-test, post-test, and 3-month follow-up surveys Measures: Self-rating of comfort in performing clinical tasks Case-based knowledge questions

23. Lessons Learned Can be done but getting follow-up surveys back is a challenge Preliminary results are encouraging � self-reported practice comfort and case-based knowledge questions appear to work as measures Survey needs to be minimum length Dropped knowledge questions in post-test because they were too soon after baseline � post-test focuses on feedback on training Nature of Level III varies: intensity/length, profession trained, topics covered, etc. Developed special versions for nurses and HepC 40 surveys collected with revised instruments this year � still working on getting all follow-up surveys back

24. Measuring Training Outcomes Through Qualitative Interviewing TX/OK AETC Women & HIV Symposium (JG) and Asilomar Faculty Development Conference (RV) Jennifer Gray, RN, PhD (JG) Richard Vezina, MPH (RV) TX/OK AETC Pacific AETC

25. TX/OK AETC Women & HIV Symposium (JG) First time region-wide symposium Multidisciplinary planning committee Lack of knowledge about gender-specific care Increased # of HIV infections among women in the region. Symposium goal: Improved care of HIV+ women Annual region-wide training conference 125 Participants, all PAETC faculty and program staff Conference goals: Improved skills and knowledge among faculty/trainers Improved training outcomes throughout region as a result of staff development

26. Evaluation Plans JG Email one month post to all registrants Simple open-ended questions, for all disciplines Identify how content was used with patients and shared with peers. RV Post-Post: Form A: Self-assessment at end of Conference Identify skills and content learned, areas in which to integrate new skills and content Form B: 6 month Follow-Up Individualized telephone interviews, reviewing Form A Focus on how skills/content were applied; barriers

27. Why these evaluation methods? Able to assess at multiple levels (Kirkpatrick model): Level 2 (Learning: improved knowledge) (RV) Level 3 (Behavior: change in practices) (JG, RV) Seeking specific content regarding conference (RV) Limited resources and time (JG) No existing tool found that met needs (JG)

28. Findings Major Themes: (RV) Identified high need for continued skills training Transferred new skills/information to coworkers and employees Barrier to continued integration: Time constraints Major Themes: (JG) Impact on patients 13 had taught patients information learned at the symposium 3 had used info for referrals 3 system changes- i. e. assessment forms, clinical strategies Shared information with others: 8 informally, 1 structured, 4 created materials Most common topics: medication/adherence, HIV in general

29. Strengths & Challenges of Methods What went well: Announced at end of symposium/conf. (JG, RV) Brief instrument encouraged higher response (JG) Longer instrument yielded rich responses (RV) What�s Next: Provide Incentives (JG, RV) Change instrument Shorter, easier instrument for higher response rate (RV) longer instrument for greater depth (JG) More effective confirmation of contact information (JG, RV)

30. Intensive On-site Training Evaluation: A Mixed Methods Approach  Debbie Isenberg, MPH, CHES Margaret Clawson, MPH Southeast AETC Intensive On-Site Training (IOST) Involves training, consultation, technical assistance and information dissemination (Levels I-V) Targeted towards new Ryan White Title III and other rural health sites Central office-based clinical instructor spends a half day to a full day at the site Intensive On-Site Training (IOST) Involves training, consultation, technical assistance and information dissemination (Levels I-V) Targeted towards new Ryan White Title III and other rural health sites Central office-based clinical instructor spends a half day to a full day at the site

31. Study Overview Main research questions Process and Impact (Reaction and Learning) What was the quality of the training? How well were learning objectives met? What are the trainees� intentions to change their clinical practice? Outcome (Learning and Behavior) How has the provider�s experience in the clinical training program impacted his/her ability (if at all) to provide HIV quality care to PLWH?

32. Study Protocol Phase One Post training CQI form completed by participants Phase Two Recruitment packets mailed 3 months after last IOST Research staff contact potential participants 1 week later for interview Phase Three Reminder letter for 2nd interview sent 9 months after initial interview (total 12 months post IOST) Research staff contact participants 1 week later for interview Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time

33. Content: Phase Two and Three Written Demographic Assessment (PIF+) Semi-Structured Phone Interview (Tape recorded) Quantitative: participant asked to rate the effect of training in each specific training area Qualitative: participant asked to give concrete examples of how training has affected their skills in the clinical area If no effect reported, participants are asked for more explanation Topic Areas:Provider knowledge and ability Identification of risk behavior, signs and symptoms of early and advanced HIV disease Viral load, CD4 counts, and treatment implications Comorbidity of substance abuse and mental illness PEP and PHS Guidelines Topic Areas:Provider knowledge and ability Identification of risk behavior, signs and symptoms of early and advanced HIV disease Viral load, CD4 counts, and treatment implications Comorbidity of substance abuse and mental illness PEP and PHS Guidelines

34. Strengths and Challenges

35. Lessons Learned Think about what motivates the training audience to participate in the study when deciding on study design Develop the protocol to lower respondent form and time burden Don�t be afraid to change the protocol midway in the study if not working Consider the resources that you have to collect and analyze the data in choosing a study design

36. Analysis of HIV Patient-Provider Communication Bradley O. Boekeloo, Ph.D., Sc.M. University of Maryland

37. Methods Providers Randomized (n=8) Brief cultural competency training vs. none Audiotapes of HIV Visits (n=24) 3 patient visits tape recorded per physician. Tapes transcribed. Patient Exit Questionnaire (n=24) Interviewer read patient questions and patient answered on an answer form.

40. Hypothesis Based on Exploratory Data and Next Steps Brief Intervention not enough for change Patients may be more comfortable discussing medical therapy than personal risk behaviors Try to determine whether different types of communication on audiotapes account for differences in patient comfort communicating with physician.

41. Presenter Contact Information NY/NJ AETC: Mari Millery, PhD 212-305-0409 mm994@columbia.edu Delta Region AETC: - Cheryl Hamill, RN, MS, ACRN 601-984-5552 chamill@medicine.umsmed.edu - Nancy Showers, DSW 732-603-9681 njshowers@aol.com Southeast AETC: - Margaret Clawson, MPH 404-712-8448 mclawso@emory.edu - Debbie Isenberg, MPH, CHES 404-727-2931 disenbe@emory.edu

42. Conference Call Evaluation Call 8: July 27, 2004 http://www.ihi.org/feedback/survey.asp?surveycode=AETCCall072704 Survey Code: AETCCall072704 For assistance contact: Lorna Macdonald at lmacdonald@ihi.org