Outcomes Assessment 101

1. Outcomes Assessment 101 Assessment Showcase 2009: Best Practices in the Assessment of Learning and Teaching February 11, 2009 Gwendolyn Johnson, Ph.D.Deborah Levine-Donnerstein, Ph.D.Assessment Coordinating Council

2. Agenda for Today • Benefits of Assessment • Procedure for Assessing Student Learning • Learning Outcomes Statements • Commonly Used Measures of Student Learning

3. Definition of Assessment “Assessment is a means for focusing our collective attention, examining our assumptions, and creating a shared academic culture dedicated to continuously improving the quality of higher learning. Assessment requires making expectations and standards for quality explicit and public; systematically gathering evidence on how well performance matches those expectations and standards; analyzing and interpreting the evidence; and using the resulting information to document, explain and improve performance.” Angelo, Thomas, “Reassessing Assessment,” AAHE Bulletin, Volume 47/Number 8, April, 1995. This definition of assessment was adapted by the University of Arizona.

4. Effective Assessment • Assesses what matters most – in what we actually teach and in what we expect our students to learn • Provides information for improving learning (and for improving teaching and assessment) • Focuses on process as well as on inputs and outcomes • Actively involves teachers and students (and appropriate others) • Uses multiple and varied measures • Is carried out at various key points • Provides feedback to those most affected Adapted from work of T. A. Angelo – University of Miami – June 1998

5. A Procedure for Assessing Learning Outcomes (1) Determine desired learning outcomes (A) Get input from stakeholders: faculty, students, alumni, employers, parents, etc. (B) Review statements from professional organization; peer institutions (B) Prioritize outcomes; decide on a schedule for assessing particular outcomes (C) Find out what is already being assessed; revise/adapt as needed. (D) Investigate existing measures not currently in use (2) Select multiple measures for each outcome being measured (E) Develop measures as needed (F) Determine sample size, schedule of administration, etc Set standards; Benchmarks (G) Identify relevant externally defined criteria (e.g. what do peers do?) (3) Carry out assessment; tally and analyze results (4a) Use results to improve program (4b) Use results to fine-tune the assessment process OIRPS Johnson 9/5/08 Adapted from work by E. Berman 1999

6. Learning Outcome Statements Collectively created and developed across the program, learning outcome statements: • Describe what students are able to demonstrate in terms of knowledge, skills, and attitudes upon completion of the program • Rely on concrete verbs such as “define,” “apply,” or “analyze” • Map to the curriculum and educational practices that offer multiple and varied opportunities for students to learn • Ideally, are collaboratively authored and collectively accepted • Incorporate or adapt professional organizations’ outcomes statements if they exist • Can be quantitatively and/or qualitatively assessed during students’ undergraduate or graduate studies

7. Learning Outcome Statements • Students will be able to:Distinguish and articulate major cultural differences between the target culture and their own. • Analyze how specific historical and cultural contexts shape particular perceptions, practices, and products of individuals • Interpret mathematical models and represent information symbolically, visually, numerically, and verbally. • Write persuasive documents that provide appropriate and effective evidence for various audiences, situations, and purposes. • Demonstrate knowledge of the formal and informal structures and processes that make social systems, governments, and economies work. • Design experiments, generate and analyze actual data, use abstract reasoning to interpret these, and formulate and test hypotheses with scientific rigor.

8. A Procedure for Assessing Student Outcomes (1) Determine desired learning outcomes (A) Get input from stakeholders: faculty, students, alumni, employers, parents, etc. (B) Review statements from professional organization; peer institutions (B) Prioritize outcomes; decide on a schedule for assessing particular outcomes (C) Find out what is already being assessed; revise/adapt as needed. (D) Investigate existing measures not currently in use (2) Select multiple measures for each outcome being measured (E) Develop measures as needed (F) Determine sample size, schedule of administration, etc (G) Consider level of incoming students with regard to benchmark (G & H) Set standards; Benchmarks (H) Identify relevant externally defined criteria (e.g. what do peers do?) (3) Carry out assessment; tally and analyze results (4a) Use results to improve program (4b) Use results to fine-tune the assessment process OIRPS Johnson 9/5/08 Adapted from work by E. Berman 1999

9. Commonly Used Measures of Student Learning • Direct Indicators of Learning • Pre/Post Tests • Course -Embedded Assessment (papers, assignments, tests) • Standardized Tests (national, accreditation) • Portfolios • Capstone Courses • Thesis/Dissertation Evaluation • Observation of Performance

10. Commonly Used Measures of Student Learning • Indirect Indicators of Learning • Student surveys • -questionnaires -interviews and focus groups • External Reviewers • Alumni surveys • Employer Surveys • Curriculum and Syllabus Analysis

11. A Procedure for Assessing Student Outcomes (1) Determine desired learning outcomes (A) Get input from stakeholders: faculty, students, alumni, employers, parents, etc. (B) Review statements from professional organization; peer institutions (B) Prioritize outcomes; decide on a schedule for assessing particular outcomes (C) Find out what is already being assessed; revise/adapt as needed. (D) Investigate existing measures not currently in use (2) Select multiple measures for each outcome being measured (E) Develop measures as needed (F) Determine sample size, schedule of administration, etc (G) Consider level of incoming students with regard to benchmark (G & H) Set standards; Benchmarks (H) Identify relevant externally defined criteria (e.g. what do peers do?) (3) Carry out assessment; tally and analyze results (4a) Use results to improve program (4b) Use results to fine-tune the assessment process OIRPS Johnson 9/5/08 Adapted from work by E. Berman 1999

12. A Program Assessment Example

13. The Important Questions to Consider: • Do the student learning outcomes match current-established goals? • Are the current outcomes what we intend to measure and assess? • Overall, are students acquiring our expected: skills, knowledge, attitudes and motivation to succeed in their academic experiences? • What changes are needed to address gaps in assessing student learning? • How can we proceed to identify shortfalls between curricula and learning outcome goals at each level?

14. A Procedure for Assessing Student Outcomes (1) Determine desired learning outcomes (A) Get input from stakeholders: faculty, students, alumni, employers, parents, etc. (B) Review statements from professional organization; peer institutions (B) Prioritize outcomes; decide on a schedule for assessing particular outcomes (C) Find out what is already being assessed; revise/adapt as needed. (D) Investigate existing measures not currently in use (2) Select multiple measures for each outcome being measured (E) Develop measures as needed (F) Determine sample size, schedule of administration, etc (G) Consider level of incoming students corresponding to benchmarks (G & H) Set standards; Benchmarks (H) Identify relevant externally defined criteria (e.g. what do peers do?) (3) Carry out assessment; tally and analyze results (4a) Use results to improve processes (4b) Use results to fine-tune the assessment process OIRPS Johnson 9/5/08 Adapted from work by E. Berman 1999

15. General Considerations • How can the assessment process be sustained? • Can efficiency in collecting, reporting, and using results be increased in all relevant areas of campus? • Should we change the standards, criteria, benchmarks, etc., on an ongoing basis to meet internal and external constraints facing universities? • In what ways must campus policy support and reflect assessment goals for student learning?

16. Sustainability: • Assessing Learning Outcomes in a Changing Academic Environment • Within the process of restructuring colleges, programs and departments, a campus retains leadership and gains new leaders to: • continue to enhance the goals of assessing student learning, • reshape and modify procedures for assessing student performance throughout this process, • reestablish linkages among new and existing units to better inform: • what is important to assess, • how to address changes in programs and curricula, • develop effective measures to assess new goals and • determine how best to use these outcomes in the classroom, department and college for greater student retention and success

17. References Angelo, Thomas. “Reassessing Assessment,” AAHE Bulletin, Volume 47/Number 8, April, 1995. Maki, Peggy L. (2004). “Assessing for Learning: Building a Sustainable Commitment Across the Institution”. Sterling, VA: Stylus PublishingWalvoord, Barbara (2004). “Assessment Clear and Simple”. San Francisco, CA: Jossey-Basshttp://www.provost.wisc.edu/assessment/manual/manual2.html

18. Contact Information Gwendolyn Johnson, Ph.D. Director of Assessment, Office of Institutional Research and Evaluation Chair, Assessment Coordinating Council gwj@u.arizona.edu 520 621-7726 http://assessment.arizona.edu Deborah Levine-Donnerstein Department of Educational Psychology