Strategic Planning through Assessment

1. Strategic Planning through Assessment A Process of Systematic Inquiry

2. Presenter Information & Qualifications

3. Objectives of this Workshop • To provide and explain a model of the process of Systematic Inquiry (Blimling & Whitt, 1999), which begins with strategic planning throughassessment • To provide an example program recently assessed, from the perspective of the residence hall director leading the program, along with her reflections using the Model • To guide you through the planning phase of systematic inquiry via the assessment lens, using either a sample case scenario we provide or using a program you are conducting or wish to conduct on your campus • To connect our focus on assessment as a lens for planning to being leaders in your profession, summarizing key points to guide you in future program development and implementation

4. The Scope: Outcomes-Based Programs A series or set of explicitly related activities, events, and experiences that are created with a specific outcome or set of outcomes in mind, and are delivered under the assumptions that the following are true: • Students will learn/gain something we deem worthwhile from the experience(s). • While some may be fun experiences, that is not their primary purpose. • Formal determination of the degree of effectiveness reported. Possibly benchmarked over time. • Decisions (financial, staff, training, etc.) are likely to be made based on the outcomes of the program(s).

5. Assessment as we know it

6. Current status of assessment, as we see it • Overemphasis on assessment as a tool or practice and absence of assessment as a perspective. • Strict focus on assessment solely as a requirement or a priority, sometimes with support, sometimes not. • Almost solely limited to being thought about and treated as a step or series of steps near or at the end of conducting a program. • Under these conditions, it is difficult to lead using assessment.

7. Common Assessment Models

8. A “Re-Visioning” of Assessment

9. Our Premise Assessment is most useful and effective when we use it as a lens through which we view and conduct strategic planning, and more broadly, when it is treated as integral to an even larger process, Systematic Inquiry.

10. So what does the assessment “lens” do? • It invites us to consider how we measure success, effectiveness, etc. throughout the process of systematic inquiry. It keeps us focused. • It asks that we begin at the end, putting outcomes before action. • It helps us to prioritize inputs and directs us toward possible training, staffing, and other needs. • It compels us to develop good planning habits and goal-oriented habits of mind. • It moves us toward a higher standard for what we produce. • It requires us to be intentional in our efforts and enables us to support or justify our decisions at any point in the process.

11. Assessment as an afterthought - a tool Assessment as a forethought - a perspective

12. Comparing the lines of thought – tool vs. persp. Recycling is important. I wanted to do something with recycling in my residence hall. So this year, we’ve been putting up posters about recycling, we added clearly marked recycling bins for plastics and aluminum cans near the trash receptacles, and my RA’s wear “GO GREEN” T-Shirts once a month . I want to see how successful this program has been. • Our school values sustainability. Students arrive at college with various levels of understanding and various degrees of previous practices of sustainability. One way to understand and practice sustainability is through recycling. • The large majority of students who are in my hall will be able to describe the importance of recycling after living here for one year. • The large majority of students will practice recycling after living here for one year. • My residents will have a substantially greater awareness and will practice to a much greater degree than residents of other halls.

13. Toward Systematic Inquiry

14. What is Systematic Inquiry? Systematic Inquiry is the “process of an intentional, organized, and ongoing search for [actionable] information”. Begins with - examination and understanding of the extensive body of research and scholarship about relevant college outcomes and learning environments. Continues with - the application of this and other knowledge to strategic planning, mindful program implementation, and appropriate data collection and analysis. Concludes by - ultimately allowing one to determine the impacts of programs or services on student learning and/or development. (compiled from excerpts of p.94-96, Blimling & Whitt, 1999)

15. The Process of Systematic Inquiry

16. Starting at the End • The process of systematic inquiry can be narrowed down to phases through which a series of questions are asked and answered. • It emphases outcomes before action. Identify what you wish to accomplish and why you wish to accomplish it. • Begin with your school’s mission, values statement. Tie these to CAS Standards and utilize the FALDOS to identify learning outcomes, measures, etc.. • It is at this pointthat you can begin to seriously consider what all the inputs must be.

17. Phase I: Strategic Planning • Methodical • Intentional • Time-bound • Decisions clearly documented • Measures are either created or obtained • Data collection and analytic methods are determined • Program delivery systems established This is all completed well before the program has begun

18. Viewing Planning through an Assessment lens In order to obtain answers to the degree or in the form which you desire, you must first begin by learning to ask the right questions in the right way. • Begins by asking many relevant questions • Continues by narrowing the list down to the most important questions • Drives the measures you will use to answer those questions • Grows into an explicit logic model, grounded in experience and research • Follows appropriate implementation guidelines, founded on clear principles • Utilizes methodology and analytic methods appropriate to the situation • Acknowledges any and all assumptions made or biases present

19. The No Woman Left Behind Program

20. The Process of Systematic Inquiry: NWLB No Woman Left Behind (NWLB) is a University Housing sponsored initiative and receives funding from outside support. The program aims at not only informing but also changing college students’ behavior when alcohol is involved. The passive programming aspect of the program was assessed. The five primary objectives for this program are: • Increase students’ awareness of sexual assault and bystander intervention through campaign materials, events, and public relations • Provide resources regarding sexual assault and bystander intervention • Identify opportunities to present programs on sexual assault, facilitated rape and bystander interventions • Serve as a campus resource • Provide young women at the University of Arkansas leadership and development opportunities

21. The Process of Systematic Inquiry: NWLB • Original Assessment Purpose: To assess the effectiveness of current NWLB marketing efforts • There was no baseline as to the number of students who know what our program is • Allows us to be more effective in the programming that we do for next year • Assessment Process • Poll 475 from a stratified random sample of 4463 residents • Volunteers go to the student rooms to collect data • Data are entered by the NWLB coordinator • Data are analyzed by Assoc. Dir. for Assessment

22. The Process of Systematic Inquiry: NWLB

23. The Process of Systematic Inquiry: NWLB • Actual Assessment Purpose • What percent of residents recognize the program with the cues provided? • What degree of understanding of the program exists among those who state they are familiar with the program? • Does the presence or absence of recognition differ by gender? • Among males and females who are familiar with or stated they recognized the program, which gender is more likely to have the greatest level of understanding? • Comparing locations where NWLB was most heavily promoted with those where promotion was only mildly promoted, are residents living in locations with heavy promotion more likely to have a greater level of understanding than those living in locations with mild promotion?

24. Results of NWLB Passive Programming Cue Recognition • Nearly half of those surveyed (46%) recognized neither of the two cues (the poster or the bracelet). • Slightly less than one-third (30.3%) recognized both cues. • About a quarter (23.7%) recognized at least one cue. • The large majority (88.4%) of those who had seen or heard of NWLB, did so through only one venue (e.g. Hall Poster, Facebook, Greek Life, Other) Degree of Understanding • Among those who recognized at least one cue, nearly half of those surveyed (47.1%) had a Poor or No understanding of the purpose of NWLB. • Less than one-fifth of them (17.3%) had Excellent understanding. Intensity of Programming • Where more intense programming was present, a larger percent had an Excellent or Good level of understanding than those in other halls (62.5% vs. 44.6%), however, the difference was not statistically significant. Effect size was 0.23.

25. Results of NWLB Passive Programming: By Gender Cue Recognition • Females were significantly more likely to recognize one or both cues than their Male counterparts (69.6% vs. 31.0%) Degree of Understanding • While not a significant difference, a larger percentage of Females had Excellent or Good understanding vs. their Male counterparts (39.0% vs. 11.1%)

26. Developing an Assessment-Informed Perspective ACTIVITY TIME

27. How Does All This Apply to Leadership? To begin moving toward Planning through Assessment, one must be willing to take the lead by exhibiting the characteristics associated with such a mindset, and a culture. • Attitude – displaying a positive attitude toward and espousing the value of systematic inquiry. The more you read about and do assessment, the easier this becomes. • Behaviors – leading by example. Bringing up pivotal questions in the very first meeting can change the trajectory of the entire project. Become familiar with the Systematic Inquiry cycle. • Advocating for Policies – Constructing clear, vetted policies that help to insure that an assessment “lens” is being generated and used to plan, implement, and report on outcomes-based programs. • Values – communicating the key role systematic inquiry plays in establishing or maintaining your own or your organization’s integrity. • Commitment – sticking to and speaking from an assessment perspective, and continuing refer back to purpose and measurement in subsequent meetings, no matter what.

28. Summation • Student learning is the overall goal. Systematic Inquiry is not only about how to get there but also leaving a map for others to follow. • Specifically defined outcomes, justified by solid reasoning, have to come before any other planning steps. • Systematically asking the right questions before you begin a new project or program will broaden your vision of what it’s going to take for your efforts to have a high probability of being successful. • Programs already in existence should be re-examined using the process provided today, or something similar in rigor and standards. • Document, Document, Document!

29. References and Resources Blimling, G.S. & Whitt, E.J. (1999). Good Practice in Student Affairs: Principles that Foster Student Learning. Jossey-Bass, San Francisco, CA. Council for the Advancement of Standards in Higher Education (2012). CAS professional standards for higher education (8th ed.). Washington, DC. Council for the Advancement of Standards in Higher Education. (2006). Frameworks for Assessing Learning and Development Outcomes (FALDOS). Washington, DC. Keeling, R.P., Wall, A.F., Underhile, R. & Dungy, G.J. (2008). Assessment Reconsidered: Institutional Effectiveness for Student Success. [United States]:ICSSIA.