Administrative Unit Assessment: Different Purposes, Different Measures

1. October 4, 2012 Administrative Unit Assessment:Different Purposes, Different Measures Elaine W. Kuo, College Researcher, Foothill College Mallory Newell, College Researcher, De Anza College Robert Pacheco, Dean, MiraCosta College Gail Shirley, Planning Coordinator, MiraCosta College

2. Resources Documents and Resource Links are available at: http://ssscauosession.wikispaces.com/

3. Today’s Topics Administrative Unit Assessment • Administrative Unit Objectives/Outcomes • Assessment • Measures/Methods • Program Review and AUOs • Best Practices • Examples, Models • Designing AUOs

4. Group Dialogue – 5 Minutes • Talk to someone next to you • Discuss where your colleges are in the AUO process • Share with all attendees

5. What is an Administrative Unit? Foothill and De Anza Colleges A unit, department, program or office that provides services to any individuals in order to directly or indirectly maximize student success. MiraCosta College A department/program providing operational services and organizational support to assist the district in advancing its Mission Statement.

6. What are Outcomes? • The results, consequences, or impact of a course of action What are Outcomes Statements? • Specific, focused, measurable written statements that • identify intended results, • specify activities undertaken, and • may specify target groups if applicable What does an Outcomes Statement look like? • In its simplest form: • [intended result] through [actions] for [target group, if applicable]

7. We know about Student Learning Outcomes, but . . . AUOs? AUOs? AUOs?

8. ACCJC and AUOs ….. ACCJC does not define what an AUO statement is, should look like, or how to assess it, yet….they are still recommending that colleges define and assess their AUOs: “To meet the standard at the level of proficiency by 2012, the team recommends that the college accelerate the implementation of the SLO, SSLO and AUO assessment cycle at the course, program and institutional levels” - ACCJC Recommendation, De Anza College

10. What is an AUO? Generally speaking . . . An Administrative Unit Outcome Statement is a concrete, measurable statement expressing the expected achievement level (target) of a specific action. Using the identified Assessment Method, actual achievement will be determined by comparing the resulting data to the expected achievement level (target). Assessment results inform Program Review, during which the Administrative Unit: • Analyzes the data • Reviews and Reflects • Develops a plan(s) in response to the data

11. What is the purpose of AUOs? • Create a tie to the college mission, core values, institutional goals, major planning documents, and/or division/department objectives • Specify what the unit intends to accomplish • Produce assessment data to determine if performance matches learning or service/operational expectations • Inform Program Review

12. How are colleges applying AUOs? The answer may differ among colleges. At Foothill and De Anza Colleges, AUOs are written statements about what clients will experience, receive understand or learn as a result of a given service. The term “Client” refers to anyone who receives the service. At MiraCosta College, AUOs are Administrative Unit Objectives that are specific, S.M.A.R.T. benchmark statements with Outcome Statements that express the expected achievement level (target). They focus on the operational services and organization support the unit provides.

13. How we define AUOs at our campuses……. The colleges we represent defined AUOs differently in ways that fit the framework of our institutional planning and program review processes: 1. Learning based outcomes 2. Service/operational based outcomes However, it is important to remember that the essence of assessment remains the same: self-reflection, evaluation, plan, implementation

14. Foothill & De Anza A learning based example: Institutional Research: After working with Institutional Research staff on projects requiring at least 2 hours of discussion, Faculty and Staff at FHDA will be able to:· Interpret and draw correct conclusions from a cross tabulation of descriptive statistics such as course success rates by ethnicity.

15. Foothill & De Anza Examples of Assessment A learning based example: The Institutional Research and Planning Office administered a survey to all data users in the fall by emailing a link to an assessment survey. The results will be discussed at a department meeting for improvement purposes and changes in services provided. http://irp.fhda.edu/cgi-bin/rws5.pl?FORM=FHDA_AUO

16. Additional examples of assessment

17. MiraCosta Model  Institutional Mission Institutional Goals (in CMP) Institutional Objectives (in Strategic Plan) Action Plans, based on Inst. Objectives Division Objectives Division or Dept. AUOs (obj. with outcome stmt.) Establish measurable assessment and timeline for each AUO Assess, evaluate at end of year ↓ ↓ ↓ Results inform Program Review CMP: A 10-year Plan Strategic Plan: 3 Plans of 3 years each, incremental Action Plans: Annual, incremental

18. MiraCosta’s current format (with future plans to migrate all to SharePoint)...... Administrative Unit Objectives: specific, S.M.A.R.T. benchmark statements written at the department/program level that advance the Administrative Unit in achieving broader Division Objectives, Action Plan Tasks (in the Strategic Plan), and, ultimately, the Institutional Objectives and Goals that guide the District in advancing its Mission Statement. Administrative Unit Outcome Statement: A concrete, measureable statement expressing the expected achievement level (target) of an Administrative Unit Objective. Assessment Results Compared to Outcomes Statement: gathered, documented information and data indicating the outcome of an Administrative Unit's actions are compared to the target set by the Administrative Unit's Outcome Statement. The results of this comparison indicate the status of an Administrative Unit's performance and whether or not changes need to be implemented in order to improve performance. Assessment Schedule (Timeline): specifies the timeframe in which a plan/course of action will be assessed and who will administer the assessment. Assessment Method (Means of Assessment): the specific tool identified and used to gather information and data that track progress toward and/or achievement of an Administrative Unit Outcome Statement. The assessment method should match the intended outcome. There are "Direct" and "Indirect" assessment methods. Direct assessment methods are quantitative (examples: timeline, usage report, error rates, cost, productivity, efficiency, tests, rubrics); Indirect assessment methods are qualitative (examples: panel interviews, questionnaires & surveys, focus groups, observations, perceived satisfaction, perceived timeliness, perceived capability).

19. Example of a service/operational based AUO statement & assessment related to the MiraCosta Strategic Plan......

20. Example of an operational basedAUO objective, statement and assessment....

21. Writing AUOs

23. Writing an AUO • Use the SMART model: • Specific: What is your unit’s intended outcome? • Measurable: Can it be assessed effectively? • Attainable: Is it reasonable? • Relevant: Does it serve the needs of the college? Does it align/support the institutional goals? • Time-bound: Is the time-frame clearly defined, with specific start and target dates?

24. WRITING EXERCISE - USING THE FOLLOWING STEPS, WRITE: An Administrative Unit Outcome Statement Handout

25. Step 1 • Select an Administrative Unit and identify one service it provides. Step 2 • Identify one objective or goal for the unit, and write a statement that specifies the intended outcome (target). • (hint: phrase the outcome as if it already has occurred.)

26. Step 3 • If using the learning based model, what should people gain from this administrative unit? • Knowledge • Skills • Attitude • Behavior • If using the service/organizational support model, consider • Effectiveness • Efficiency (cost savings measures, turn-around time, improving processes) • Level or volume of activity • Compliance with external standards/regulations • Client satisfaction

27. Step 4 • Use Active/Action verbs • Writing statements with active/action verbs • Concrete • Refer to a process • Produce statements that are measurable Provide Implement Identify Collect Investigate Modify Complete Schedule Select

28. Step 4 (continued) • Avoid using static verbs • State-of being or condition verbs • Produce statements that are difficult to measure reliably Like Want Believe Know Feel Think Need

29. WRITE

30. Assessing AUOs

31. Assessment • Assessment Timeline • What to measure? • Select AUOs to assess on a regular cycle • When to measure? • Depending on each institution’s cycle • Who will measure? • Assign tasks • Resources needed

32. Assessment Methods • Assessment Methods • Direct (Quantitative) – examples: • Timeline • Error rates • Productivity • Cost • Indirect (Qualitative) – examples: • Questionnaires/surveys • Interviews • Focus Groups • Observations

33. Assessment Strategies • Involve all stakeholders • Work with what already exists • Design efforts with assessment in mind • Collect data from a targeted sample • Work with other units that have similar outcome statements

34. WRITING EXERCISE:Write an Assessment Methodand anAssessment Timelinefor yourAUO Handout

35. EVALUATION

36. Evaluation The process of reviewing assessment results (data) against the stated criteria (the outcome statement). Purpose: Prepare for Program Review

37. PROGRAM REVIEW

38. Program Review To Recap: • Assessment results inform Program Review • Purpose: • to draw data-based conclusions that guide future action

39. The End Questions? Documents & Resource Links are available at: http://ssscauosession.wikispaces.com/