Empowering Assessment: Building a Culture of Evaluation in Programs

1. An Adventure in Assessment Stephen “Gavin” Weiser, M.Ed stephen.weiser@gmail.com

2. Let’s Hear About You • Find someone from a similar type program as folks are talking.

3. Outline • What is Assessment • Purposes of Assessment • Some Pitfalls • Methods • Learning Outcomes - Creation • Tool Selection & Creation

4. What is assessment? • A. What we do to people we don’t like • B. An old wooden ship • C. Something we all hate and loathe • D. A sustainable process of collecting data to improve upon services & programs

5. Purposes • To collect data in order to better understand what we are doing • What can this look like?

6. Pitfalls • “teaching to the test” • The politics • Fatigue – particular to certain methods • Some bad ways to use assessment.

7. Methods • Qualitative vs. quantities • Mixed Methods • Literature review

8. Learning Outcomes • Why are they important? • What are they? • Measureable & meaningful • Examples: • To gain leadership abilities. • Good or bad?

9. Tool Selection • Based upon what we are trying to measure • Different Types: • Surveys • Focus Groups • Interviews • Structured vs. Unstructured Protocols • Field Observation

10. Create a“Culture of Assessment” • What does this look like? • Vital to it’s and the programs success!

11. Example Project

12. Background of Outdoor Recreation at Carolina • Opened Fall 2008, Director: Katie Coley • Information from Katie, annual report, focus groups with current student leaders • Annual Report • Strengths • Weaknesses • 5 year plan

13. Programs of Outdoor Recreation • Climbing Wall • Clinics • Adventure Trips • Bike Shop • Student Leadership • Outdoor Leadership Training

14. Focus Groups • 8 Students Leaders; 2 groups of 4 • Methodology • Provided information based on experiences and goals with outdoor recreation • Information received assisted in guiding what learning outcomes should be created

15. Previous Assessments in Outdoor Recreation • Office had “learning outcomes” but they were immeasurable • Same tool was used to assess all programs • Tool focused on satisfaction

16. Purpose of Assessment • New office which allowed for flexibility • To create a process that can be used for future assessments within Outdoor Recreation • Outdoor Recreation programs nationally have a need for quality assessments • Provide feedback on some of the learning outcomes related to the Rock Wall

17. Methodology • Students that have completed climbing the rock wall • University 101 and walk-up participants • A qualitative assessment was distributed post wall climb • 2 week period in November – only one week is included in this analysis

18. Learning Outcomes • Started from focus group, Katie’s feedback and goals • Took two goals: Teambuilding and Trust and developed outcomes from them • Created measurable outcomes

19. U101 Outcomes • Recognize individual’s strengths within a working group. • Understand how to apply strengths in an existing challenge. • Demonstrate a willingness to rely on others during a challenge. • Participants will learn technical skills related to the activity in which they participate.

20. Individuals’ Outcomes • Demonstrate an increase in willingness to go beyond limitations and boundaries. • Participants will learn technical skills related to the activity in which they participate.

21. Tool • Selected a qualitative instrument based on: • feasibility, rationale, design, research questions • Technically mixed method • Local instrument – developed by us • Open-ended questionnaire • Challenge: probing but not leading and receiving the information that was necessary • Katie’s feedback to make improvements to the tool • Created a second tool

22. Analysis • Used a coding procedure • “Focused coding” • Codes were guided by outcomes and data • U101: coded as whole system • Individual: coded by question • Used SPSS to develop codebook and pie charts for representations

23. Results • Solid process of developing a survey based on targeted outcomes – Major Result • Most applicable to outcomes will be discussed • Others are informative but not directly tied to outcomes

24. Individual: Expectations and Challenges

25. Skills Learned Individual U101

26. U101: How were challenges overcome? Interesting to note, this came as a surprise

27. Points of Improvement • Tool did not always provide needed data • Not a large sample size • Difficult to know if technical skills were learned during activity • Time constraints

28. Strengths • Informed by focus group data • Process for developing outcomes and tool was well documented and could be replicated • Tool design – well-sequenced • Informed support from whole office • Code and analysis were manageable and clear

29. Recommendations for Future • Use process to develop a comprehensive tool for each activity • Connect outcomes to Campus Recreation outcomes • Use satisfaction data and skills data to publicize the wall, attract non-climbers • Be intentional in educating student participants about outcomes • Expand office staff

30. Partner, Pair, Share • Find that special someone from earlier • Develop outcomes and the beginnings of a tool.

31. Thank You! • Contact: Gavin Weiser @ • stephen.weiser@gmail.com • (609) 458-2151