From Satisfaction to Impact: Assessing Faculty Learning and Development

1. From Satisfaction to Impact: Assessing Faculty Learning and Development 2013 POD Network Conference Pittsburgh, PA Megan Rodgers, Ph.D. student (rodgermm@jmu.edu) Cara Meixner, Ph.D. (meixnecx@jmu.edu)

2. Think about this prompt, then engage with a person you don’t know well: What are the opportunities and challenges related to assessing faculty development? Personally, how do you feel about systematic assessment of faculty development? Quick Diagnostic:Think, Pair, Share

3. As a result of participating in this interactive session, participants will: • Be able to explain the assessment cycle and the levels of assessment affiliated with Kirkpatrick’s model; • Be able to express hopes and concerns about assessment; • Have at least one idea of how assessment could positively affect their office. Outcomes

4. Moving from Satisfaction to Impact

5. Why Assessment? • Call for faculty developers to move beyondsatisfaction; • Minimal evidence in literature of rigorous faculty development program evaluation; and(Chism & Szabo, 1997; Hines, 2009; Kucsera & Svinicki, 2010) • Faculty self report data may be inaccurate. (Ebert-May, Hodder, Momsen, Long, & Jardeleze, 2011)

6. Kirkpatrick’s Model

7. Four Levels of Evaluation Group Activity Self-select into one of the four levels of evaluation. Within your group, designate a scribe and presenter. Consider an outcome appropriate to the level of evaluation you selected. In your fantasy land (unlimited resources), what evidence would you want to give to your Provost to showcase effectiveness in this area? Kirkpatrick (1978)

8. Assessment Cycle 1 7 2 3 6 5 4

9. Sharing of materials and data. Objective refinement Essence of all original objectives maintained External perspective helped clarify Missing objectives? Forums (unstructured focus groups) with jmUDESIGN graduates Inclusion of “affective” objectives • Establishing Objectives 1

10. Complete! Minor modifications • Mapping Objectives to Programming 2

11. Selecting or Designing Instrument 3 • 3 Surveys (Pre, Post, Post 2) • Aligned to outcomes • Embedded “Articulation” assignment • Facilitator used a rubric to evaluate • “Direct Measure”

12. Implementation Fidelity 4 • Is the program being implemented as planned? • With a detailed plan, you can have participants, facilitators, and/or an independent observer “check” the fidelity of the program

13. Collecting Information • Analyzing Data 5 6 • Various sources and time-points of data • Graduate student maintains and analyzes data • Used a “unique identifier” to match responses

14. Use Results 7 • Program Improvement • Fidelity data • Proof of Program Effectiveness • Can now speak about direct outcomes of the program

15. Action Planning

16. References Chism, N. V. N., & Szabó, B. (1997). How faculty development programs evaluate their services. Journal of Staff, Program, and Organization Development, 15(2), 55-62. Ebert-May, D., Derting, T. L., Hodder, J., Momsen, J. L., Long, T. M., & Jardeleza, S. E. (2011). What we say is not what we do: effective evaluation of faculty professional development programs. BioScience, 61(7), 550-558. Fink, L. D. (2003). Creating significant learning experiences. (1st ed.). San Francisco, CA: Jossey-Bass. Hines, S.R. (2009). Investigating Faculty Development Program Assessment Practices: What's Being Done and How Can It Be Improved?. The Journal of Faculty Development, 23(3), 5-19. Kucsera, J. V., & Svinicki, M. (2010). Rigorous evaluations of faculty development programs. The Journal of Faculty Development, 24(2), 5-18. Kirkpatrick, D. L. (1978). Evaluating In-House Training Programs. Training and Development Journal, 32(9), 6-9. Kirkpatrick, D.L., & Kirkpatrick, J.D. (2006). Evaluating training programs: The four levels (3rd ed.). San Francisco: Berrett-Koehler Publishers, Inc.