1 / 29

A Process Approach to Outcome Measurement

This presentation will probably involve audience discussion, which will create action items. Use PowerPoint to keep track of these action items during your presentation In Slide Show, click on the right mouse button Select “Meeting Minder” Select the “Action Items” tab

derron
Télécharger la présentation

A Process Approach to Outcome Measurement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. This presentation will probably involve audience discussion, which will create action items. Use PowerPoint to keep track of these action items during your presentation • In Slide Show, click on the right mouse button • Select “Meeting Minder” • Select the “Action Items” tab • Type in action items as they come up • Click OK to dismiss this box • This will automatically create an Action Item slide at the end of your presentation with your points entered. A Process Approach to Outcome Measurement

  2. A Process Approach to Outcome Measurement • Topics covered in this presentation • Design of the evaluation • Description of the participants • Knowledge findings • Participant learning styles • Instructor teaching styles • Intention to change practice findings • Motivation to change findings • Lessons learned OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  3. The Study • These data were developed at the 2006 Optimal Management of HIV Conference; • My collaborators, Dr. Harold Kessler and Michael Saag, and I are in the process of analyzing these data for publication; • The analyses I report here are preliminary are completed only for the purpose of these discussions; OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  4. Two: Elements : Participant Descriptors and Process Measurement OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  5. Data gathered • Learning style • Preference indicated by degree of endorsement of one or the other end of a dichotomy Example I prefer Hands-on learning experience ………………………………...Learning through thinking and reasoning • Learning through simulation …………………………………Learning through lectures OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  6. Data gathered (continued) • Demographics • Questions on age specialty and Board status delivered on the Learning Style questionnaire • Motivation to change • Motivation indicated by a modified standard scale of motivation to change Example It’s important to use the new approaches I’ve learned Strongly Agree - Strongly Disagree OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  7. Data gathered (continued) • Pre-Post Tests • Knowledge • Simple multiple choice knowledge based tests, one question per presenter per test, rotated to prevent order bias Example: Which of the following antiretrovirals is the least likely to cause DSPN • indinavir • zalcitabine • stavudine • didanosine • Laminvudine OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  8. Data gathered (continued) • Pre-Post Tests (continued) • Intent to change • Procedural intent questions provided prior and post the session Example Pre When treating HIV patients with PCP I use Strongly Agree - Strongly Disagree TMP-SMX or TMP-dapsone Post When treating HIV patients with PCP I Strongly Agree-Strongly Disagree intend to use…. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  9. Data gathered (continued) • Teaching style • 3-5 observers rated each session using simple descriptors of the presentation using agreement or disagreement with the description Example The Presentation focused on clinical application Strongly Agree - Strongly Disagree The presenter focused on the underlying science of medicine Strongly Agree - Strongly Disagree OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  10. Description of the participants • Of the total participants in the program, analysis is focused on the 62 complete datasets, these participants provided data for all three days both in the morning and afternoon; • While it is possible that they differed in some systematic way from the rest of the participants, complete data will be necessary for the ultimate analysis OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  11. Results: Participants were generally middle aged • The participants had an average age of 49; • The youngest was 29 and the oldest 73; • The median age was 47 indicating a slight skew to the younger side of 49. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  12. The average participant graduated from medical school in 1984 • The youngest participant graduated in 2003; • The oldest participant graduated in 1958 OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  13. Most participants were in infectious disease or general internal medicine OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  14. Most of the participants were ABMS Board Certificants • 62% of participants reported Certificant status for a Board; • 37% were not members of a Board or did not respond. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  15. The most frequent Boards of participants were Infectious Disease or Internal medicine OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  16. The participants improved their performance • Participants improved their performance from pre to post measures • The improvement was approximately 10%. • The improvement was significant p < .001. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  17. Participants also showed an expectation of changing practice patterns • Intention or expectation of changing practice patterns was measured by asking the likelihood of a particular practice being adopted; • The likelihood was measured prior to the session (Morning) and after the session (Afternoon). • The stated likelihood increased from 3.4 to 4 on a 5 point scale. • The effect was assessed and the increase in likelihood is significant p<.001. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  18. Learning style was assessed a modification of a scale developed by Kolb • The approach to learning style that was employed in this study was a modified version of Kolb’s learning style inventory. • We employed his core scale, adapting it from a dichotomy to a numeric format with a Kolb descriptor on each end of the scale. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  19. Teaching style was assessed by an observer assessment scale • The items were selected to reflect the teaching characteristics for the Kolb learning styles; • During each presentation, 3-5 observers assessed the style of each presenter. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  20. Motivation to change • Participants were asked to endorse 30 statements of change commitment; • The statements were modified from the original focus to reflect motivation and readiness to learn new information and make changes in clinical practice patterns. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  21. A Process Approach to Outcome Measurement • Outcome measurement • Direct assessment/measurement of actual practice change or improvement in patient health status is difficult; • Focus is on valid secondary measures such as: • Self-report; or, • Vignettes. • The core question for CME measurement professionals is the validity of such secondary measures. • Assessment validity is typically based on a combination of: • Face validity (is it reasonable); • Construct validity (does it render a measurement correlated with some other measure deemed valid); and, • Predictive validity. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  22. First we looked at the acquisition of knowledge • We used the teaching style judgment questions as indicators of an underlying set of predominate teaching styles • We then used the averaged teaching style scores for each instructor as predictors of knowledge acquisition of the participants. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  23. Teaching and Knowledge Acquisition • We looked at the relationship between Teaching Style and Knowledge: • We found a significant relationship between the two measures; • The unobserved variable Teaching Style explains approximately 25% of the variance in Knowledge. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  24. We looked at the formation of an intent to change • First, we used the learning style questions as indicators of an underlying set of learning preferences • We then used those preferences as predictors of intent to change practice patterns OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  25. Learning Style, Knowledge and Intent to Change Practice • We looked at the relationship between Learning Style, Knowledge and Intent to change practice patterns: • We found a significant relationship between the two causal measures and Intent; • The unobserved variable Learning Style explains approximately 13% of the variance in Intent; • The variable Knowledge explains approximately 55% of the variance in Intent. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  26. Self report of practice change • First we looked at the relationship between Motivation to change and reported practice change: • Motivation to change was assessed using a series of readiness for change items; • Change items were taken as indicators of an underlying state of change readiness. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  27. Motivation and Intent to Change Practice and Self-reported Practice • We looked at the relationship between Motivation to change, Intent to change practice patterns and self-report of practice patterns at a 12 month follow-up: • We found a significant relationship between the two causal measures and self-report practice change; • We also found a relationship between motivation and intent to change. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  28. A Path Model of Cognitive Elements of CME Efficacy • We looked at the relationships among a number of variables as they relate to practice change. We found: • Teaching Style has a path to Knowledge; • Knowledge and Learning Style have paths to Intent to Change; • Motivation to Change and Intent to Change have paths to Self-reported Practice Change. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

  29. Discussion • All of these variables are related in a causal manner with self-reported practice change; • Each variable has a theoretical reason to be thought of as causally related to practice change; • These data suggest that there are multiple variables that could provide a meaningful estimate of the degree of efficacy of the CME program. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

More Related