Leading (and Assessing) a Learning Intervention

1. Leading (and Assessing) a Learning Intervention IMPACT Lunch and Learn Session August 6, 2014 Facilitated By Ozgur Ekmekci, EdD Interim Chair, Department of Clinical Research and Leadership School of Medicine and Health Sciences The George Washington University

2. Overview • Leading a Learning Intervention • In today's complex social environment, anyone who has been assigned the task of leading a learning initiative has to approach the process in a systematic way in order to succeed. The crucial success factor is often creating alignment amongst components that constitute a well-articulated learning intervention proposal. This session is offered for those who would like to better understand the four key questions that allow educational leaders to effectively envision, design, implement, and assess a meaningful learning intervention, whether this intervention takes place in the classroom or in the boardroom.  • Learning Objectives • Discuss challenges associated with implementing a learning intervention • Analyze major components of an effective learning intervention proposal • Analyze four key questions that drive a learning intervention • Discuss how assessment fits into the larger context of a learning intervention proposal • Review six major assessment methods used in health sciences

3. Alignment of Major Components KEY SUCCESS FACTOR • Learning Community Context (overview of the learners’ location, background, roles, and interests) • The Problem Statement (current state of the learners – in measurable outcomes - and why this is a problem) • The Statement of Significance (why it is important that the intervention be implemented and who suffers in what way, if not implemented) • The Proposed Solution (envisioned learning objectives - including the statement of tangible, measurable objectives, which tie back to the problem statement - i.e., current state) • Implementation Time Line (a graphical depiction of the implementation plan, with major milestones along the way) • Evaluation Method(s) (how to measure progress against learning objectives and how it will be known when the learners finally get to the envisioned future state) • Conclusion (summary of key messages with which to leave the audience) ALIGNMENT

4. Learning Intervention Timeline Review and Revise Content Method or Schedule Conduct CAT Determine Effectiveness of Intervention STOP Ekmekci, O. (2013). Being there: Establishing instructor presence in an online learning environment, Higher Education Studies. 3(1). 29-38. NO Delivery Complete ? Deliver Content Conduct Formative Assessment YES Conduct Summative Assessment START Provide Learner Feedback Provide Learner Feedback Learning Intervention Time Line PHASE I PHASE III PHASE IV PHASE II • Define Measurable Learning Objectives: • Individual Level • Change in behavior • Change in perception • Change in performance • Organizational Level • Change in outcomes • Societal Level • Change in environment • What do I want to change, for whom, by how much, and by when? In other words, what will success look like? • Plan Learning Intervention: • Identify learners • Select content • Develop materials • Design delivery method • Construct learning environment • Develop delivery schedule • How will I change what, for whom, by how much, and by when? In other words, how will I achieve success? • Design Learning Evaluation: • Identify key variables to be measured at the appropriate level of analysis (i.e., individual, organizational, societal) • Decide on the most appropriate method(s) for conducting formative and summative assessments that measure the key variables • Determine frequency and timing for formative assessment and summative assessments • Plan delivery format, medium, frequency, and timing of feedback to learners • Select quantitative statistical methods that will help determine the effectiveness of intervention • How will I know what I am changing, for whom, by how much, and by when? In other words, how will I monitor progress against set objectives that define success? • Implement Learning Intervention and Conduct Learning Evaluation: • Deliver content • Assess learning • Provide feedback to learners • Analyze data using statistical methods • Determine effectiveness of learning • intervention • Declare success! • Has what I wanted to change, for those whom I wanted to change it, actually changed by the amount I had wanted it to change?

5. Reflecting on Four Key Questions Ekmekci, O. (2013). Being there: Establishing instructor presence in an online learning environment, Higher Education Studies. 3(1). 29-38. PHASE I PHASE II PHASE III PHASE IV Define Measurable Learning Objectives What do I want to change, for whom, by how much, and by when? In other words, what will success look like? By asking this question, the leader will be able to contemplate and quantify the envisioned type of change, its scope (in terms of breadth, depth, and timeline), and the level at which change is to take place (i.e., individual, organizational or societal) – which will help ability to monitor progress against objectives Plan Learning Initiative How will I change what, for whom, by how much, and by when? In other words, how will I achieve success? By asking this question, the leader will more easily identify participants, select content, develop materials, design delivery method, construct learning environment, and develop delivery schedule Design Learning Initiative Evaluation How will I know what I am changing, for whom, by how much, and by when? In other words, how will I monitor progress against set objectives that define success? By asking this question, the leader will be able to identify key variables to be measured, decide on methods and frequency for formative and summative assessment, plan delivery format, medium, frequency, and timing of feedback to participants, and select statistical methods that will help determine the effectiveness of the initiative Implement Learning Initiative & Conduct Evaluation Has what I wanted to change, for those whom I wanted to change it, actually changed by the amount I had wanted it to change? By asking this question, the leader will be able to effectively deliver content, assess progress, provide feedback to participants, and analyze data using statistical methods to ultimately determine how successful the initiative has been

6. Assessment as a Process • Assessment is the ongoing processthat allows one to: • establish clear, measurable expected outcomes of learning; • ensure learners have sufficient opportunities to achieve those outcomes; • systematically gather, analyze, and interpret evidence to determine how well learning matches expected outcomes of learning; and • using the resulting information to understand and improve learning. • Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge. p. 4.

7. Threats to Validity • Construct Underrepresentation (CU) • Not measuring the right things: • under-sampling of the achievement domain – such as tests that are too short to support legitimate inferences to the domain, trivial questions written at low-levels of the cognitive domain, or mismatch of sample to domain • Construct Irrelevant Variance (CIV) • Not measuring (the right) things in the right manner: • erroneous inflation or deflation of test scores due to certain types of uncontrolled or systematic measurement error – such as poorly crafted test questions, rater bias, or environmental conditions • Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge. 

8. Levels of Assessment and Matching Methods Behavior Observational Assessment Performance Test Simulation DOES Requires methods that provide an assessment of routine “real” performance for the learner SHOWS HOW Requires methods that allow learner to demonstrate the integration of knowledge and skills that will translate into successful routine performance Portfolio KNOWS HOW Requires methods that provide an assessment of how well learner uses knowledge in the acquisition, analysis, and interpretation of data and the development of an approach or plan Written Test Oral Examination KNOWS Requires methods that provide an assessment of how well learner has acquired the fundamental knowledge on which to build higher levels of knowledge and behavior Cognition Adapted from: Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge. 

9. Wrap Up Questions?