Program Assessment Workshop for Staff A Presentation for Palo Alto College 1/10/2018

1. Program Assessment Workshop for StaffA Presentation for Palo Alto College1/10/2018 Keston Fulcher, PhD James Madison University

2. What is Assessment? • Assessment is a bureaucratic exercise that has little bearing to our work. OR… • Assessment is a process that compellingly tells our story: The goals we’ve set; the strategies we’ve used to accomplish them; how we’ve affected our students and institution; how we’ve learned from our failures; how we’ve celebrated our successes.

3. A Note about Improvement • The Damon Dash Story….

4. Overview • Admin vs. Academic Program Asmt • Outcomes for Participants • Contrast outcomes and actions in an assessment framework • Distinguish among beginning, developing, good, and advanced assessment reporting • Discuss the fundamentals of identifying problems, collecting baseline data, systematically addressing the issues, and then re-assessing to determine success

5. Admin vs. Academic Asmt • Academic Units: Focus typically on student learning outcomes • Administrative Units: Focus typically on effectiveness/efficiency (more on this later) • Administrative Units: Much more diverse

6. Outcome vs. Action • (Intended) Outcome: A desired IMPACT of your office’s efforts • Action: What you do to achieve outcome • The two are often confused. However, sometimes the distinction is not clear cut.

7. Example 1 • Action Two pills every four hours for three days • Intended Outcome My temperature will drop from 101 to 98.6 Cause - Effect

8. Example 2 • Action I will • exercise 30 minutes four days a week and • eat approximately 2000 calories a day • Intended Outcome I will lose 20 pounds Cause - Effect

9. Example 3 • Action • Realtors will attend summer retreat on better sales tactics. • Realtors, on average, will show 50 houses per year. • The marketing office will advertise via Google, the weekly real estate digest, etc. • Intended Outcome The real estate office will sell 10 million dollars worth of homes in 2018 (2 million more than 2017) Cause - Effect

10. Outcome or Action? • The maintenance crew will keep the lawn mowed.

11. Outcome or Action? • The assessment director will meet with all unit coordinators on campus.

12. Example 4 (A PAC Flashback) The year, 2008 • Top Song: https://www.youtube.com/watch?v=dq6Q_uaJF4k • Top Movie: https://www.youtube.com/watch?v=VNKNQwJIlBc

13. Top News Story 2008 Image From: http://www.businessjournalng.com/case-study-the-collapse-of-lehman-brothers/

14. 2008 PAC FT 4-yr FTIC Graduation Rate 11.2%

15. Segue to Outcome/Action • (Intended) Outcome: The 2012 FT PAC cohort will have a 4-yr FTIC graduation rate of 22.4%. Note, 2008 rate was 11.2% • What actions did you take to get here….

16. In Business Parlance • Outcomes = LAG measures • Actions = LEAD measures

17. The Assessment Cycle

18. Step 1: State Service-Level Outcome Service-Level Outcomes (SLOs) are the desired effect your unit/program wishes to have.

19. Step 2: Map Actionsto Outcomes Identify actions you intend to take to achieve desired outcome Referred to as Program Theory • How the design of a program should theoretically affect outcomes

20. Step 3: Select Methods Select instruments and data collection strategies to gather evidence about outcomes.

21. Step 4: Analyze and Interpret Results What did you find, and what does it mean relative to your outcomes and actions?

22. Step 5: Reporting to Stakeholders Identify stakeholders and what they want and need to know.

23. Step 6: Using Results for Improvement Program/unit improvement based on evidence is the main purpose of assessment

24. O.k….Let’s Pivot to Assessment Reporting • Four hypothetical assessment reports; handing out one at a time; each progressively better • Report format inspired after PAC’s • Purpose: Aligned with…Distinguishing among beginning, developing, good, and advanced assessment reporting • Context: • Unit: The Health Center • Outcome: Flu shots ACTIVITY

25. On Learning Improvement A Pig Never Fattened Because It was Weighed

26. Fundamental Considerations of Improvement • It’s about relationships and trust • Leadership within the unit • Clarity of shared vision regarding improvement • Tight relationship between actions and outcomes • Development of staff • Assessment built around (pre and post) actions • Contextual factors

27. Practice Scenario… • Pretend you are an improvement consultant The director of the Health Center approaches you… • You are trying to size up the program and its readiness for an improvement project. • What questions should you ask? • For example, what do you want to improve? • Work with your partner to generate questions. • 10 minutes. • Let’s role play ACTIVITY

28. Summary and Questions • Outcomes for Participants • Contrast outcomes and actions in an assessment framework • Distinguish among beginning, developing, good, and advanced assessment reporting • Discuss the fundamentals of identifying problems, collecting baseline data, systematically addressing the issues, and then re-assessing to determine success

29. fulchekh@jmu.edu • References • First Pig Paper: http://learningoutcomesassessment.org/documents/Occasional_Paper_23.pdf • Second Pig Paper: http://www.rpajournal.com/return-of-the-pig-standards-for-learning-improvement/