Establishing a Culture of Assessment Evaluation of the Effectiveness

1. Establishing a Culture of AssessmentEvaluation of the Effectiveness Alabama A&M University January 07, 2008 John D. Jones, Ph.D. Associate Provost for Institutional Effectiveness and Research Bennett College

2. Goals for the Workshop • clear understanding of how to develop an assessment plan • more knowledgeable about assessment methods • more knowledgeable about analyzing and reporting assessment results • more knowledgeable about how to assure the quality of assessment plans and the reporting of assessment results

3. What Do We Mean by Assessment? • it is a formative evaluation process designed to support improvement • it is continuous • it is focused on improvement • student learning • student development • the institution and its people

4. Levels of Assessment • classroom assessment • assessment of individual students performance at the course level by an instructors • course assessment • assessment of how well a course is meeting student learning outcomes • program assessment • assessment of how well an academic program is meeting student learning outcomes • assessment of how well a support program is meeting its objectives • institutional assessment • assessment of campus-wide issues or programs

5. Institutional Effectiveness and Program Assessment • focuses on continuous quality improvement • academic programs • administrative support units • examines students learning outcomes, program outcomes, customer satisfaction, and unit performance to identify areas to improve • leads to actions • conducted annually

6. Purposes of Program Assessment • to improve • the assessment process should provide feedback to determine how the program can be improved • to inform • the assessment process should inform faculty and other decision makers of the contribution and impact of the program • to prove • the assessment process should encapsulate and demonstrate what the program is accomplishing to students, faculty, staff, and outsiders • to support • the assessment process should provide support for campus decision-making activities such as program review, strategic planning, and external accountability activities such as accreditation

7. Effective Program Assessment Should Answer these Questions • What are you trying to accomplish? • How well are you doing it? • How, using the answers to the first two questions, can you improve what you are doing? • What and how does a program contribute to the development and growth of its students and/or the support of its customers? • How can student learning be improved?

8. Program Assessment is Effective When Assessment… • is viewed as comprehensive, systematic, and continuous activity • is viewed as a means for self-improvement • measures are meaningful • utilizes multiple measures and multiple sources • is used as a management tool • results are valued, and are genuinely used to improve programs and processes • has sufficient coordination and review • includes all constituents in the process (e.g., faculty, staff, administrators, students, community, industry)

9. Mechanics of Assessment • assessment is a continuous improvement process • to improve, you need to know where you are today and where you would like to go • mission (purpose) • vision (where you would like to go) • goals (steps to getting where you would like to be) • objectives or outcomes (what you need to achieve in order to get there) • measures (how well you are currently doing) • to improve, you need to take action • analyze your program or operations to determine changes • plan the changes • take action

10. Organizing for Assessment • before assessment can begin, key players, committees and structures must be identified and assume responsibility for designing, implementing, and carrying out the assessment process • understanding the needs of program or unit can you help think about the design of the assessment plan • depending on purpose, the plan can be informal (for internal use) or formal (external audience)

11. Identify the Scope of the Plan • What should the assessment include? • assess resources (facilities, students, faculty) • assess processes (pedagogy, advising, feedback) • assess results or outcomes • who/what gives you feedback • academic program assessment typically focuses on student outcomes • administrative unit assessment typically focuses on quality of products, processes, and services

12. Defining the Mission Statement • a broad statement of what the program or unit is, what it does, and for whom it does it • a clear description of the purpose of the program or unit and the learning environment • reflects how the program contributes to the education and careers of students graduating from the program or how the unit supports its customers • aligned with department, college, and university missions • distinctive for your program or unit

13. Components of a Mission Statement • primary functions or activities of the program or unit • most important functions, operations, outcomes, and/or offerings of your program or unit • purpose of the program or unit • the primary reasons why you perform your major activities or operations • stakeholders • groups or individuals that participate in the program and those that will benefit from the program or unit

14. Structure of a Mission Statement • “The mission of (name of your program or unit) is to (your primary purpose) by providing (your primary functions or activities) to (your stakeholders).” (Additional clarifying statements) • (Note: the order of the pieces of the mission statement may vary from the above structure.)

15. Checklist for Mission Statement • Is the statement brief and memorable? • Is it distinctive? • Does it clearly state the purpose of the program or unit? • Does it indicate the primary function or activities of the program or unit? • Does it indicate who the stakeholders are? • Does it clearly support the department’s, college’s, and university’s missions?

16. Defining Program Goals • establishment of goals prior to developing objectives is recommended • some assessment processes don’t include reporting of goals • goals are long-term organizational or program targets or directions of development • they state in broad terms what the organization wants to accomplish or become over the next several years • goals provide the basis for decisions about the nature, scope, and relative priorities of projects and activities • goals should help move the organization or program to attain its vision

17. General Process Used in Generating Goals • examine mission • think what that unit or program would look like and how its services and operations (refer to you mission) would need to be conducted to reach that vision • improve, minimize, maximize, provide • generate list of potential goals and prioritize • state these as goals • two general approaches to developing goals • review existing documents • ideal unit or program approach

18. Defining Objectives and Student Learning Outcomes • objective • a measurable target with a time limit that must be met on the way to attaining a goal • student learning outcome • specific statements that describe the intended learning outcomes that must be met on the way to attaining the degree • typically derived from goal statements • more precise, more specific and measurable than the goal statement • there can be more than one objective or outcome related to each goal

19. Think SMART When Defining Student Outcomes and Program Objectives • Specific • clear and definite terms describing the abilities, knowledge, values, and attitudes • Measurable • it is feasible to get the data, data are accurate and reliable, it can be assessed in more than one way • Aggressive but Attainable • don’t let the perfect divert you from what is possible • Results-oriented and Time-bound • describe where you would like to be within a specified time period • describe what standards are expected from students or what percent of students are expected to achieve a particular level of performance

20. Agenda Identifying Student Learning Outcomes • What is a student learning outcome • Writing student learning outcomes • Types of student learning outcomes • Identifying key outcomes

21. Agenda…continued Measuring Student Learning Outcomes • Direct evidence • Indirect evidence

22. Student Learning Outcomes • What is a Learning Outcome? Knowledge, skills and attitudes that students take with them from a learning experience.

23. Writing Learning Goals • Aim for goals that are neither too broad or too specific • Use concrete action words • Define fuzzy terms • Focus on the end, not the means • Focus on the most important goals • Work collaboratively with colleagues

24. Types of Learning Goals Increase knowledge and basic understanding • Memorization • Replicating a simple procedure • Defining concepts • Summarizing concepts • Explaining concepts

25. Types of Learning Goals Thinking, Performance, and Interpersonal Skills • Analysis, evaluation, problem solving, and decision making • Physical skills related to manipulating ideas, concepts, tools etc. • Listening, working with people from diverse backgrounds, teamwork, leadership ability

26. Types of Learning Goals Attitudes and Values • Appreciation • Awareness of one’s own values, attitudes, and opinions • Integrity • Character • Valuing learning

27. Most Valued Learning goals Today! • Communication skills • Information literacy and research skills • Thinking skills • Interpersonal skills

28. Identifying key learning goals • Identify resources for potential goals • Make it a collaborative process • Achieve consensus on common goals • Preserve academic freedom • Share goals with students

29. Measuring Student Learning Outcomes • Direct evidence of student learning is tangible, visible and self-explanatory evidence of exactly what students have or have not learned.

30. Examples of Direct Evidence • Ratings of student skills by their field experience supervisors. • Scores and pass rates on appropriate licensure/certification exams. • Capstone experiences such as research projects, theses, dissertations, exhibitions, performances scored using a rubric. • Other written work or performance, scored using a rubric. • Portfolios of student work.

31. Examples of Direct Evidence… continued • Scores on locally-designed tests such as final examinations in key courses, qualifying exams, and comprehensive exams, accompanied by test blueprints describing what the test assess. • Score gains between entry and exit on published test, local test, or writing samples. • Employer ratings of the skills of recent graduates. • Student reflections on their values, attitudes, and beliefs.

32. Measuring Student Learning Outcomes • Indirect evidence of student learning provides signs that students are probably learning , but the evidence of exactly what they are learning is less clear and less convincing.

33. Examples of Indirect Evidence • Course Grades • Assignment grades, if not accompanied by a rubric or scoring guide • Admission rates into graduate school or graduation rates • Placement rates of graduates into appropriate careers and starting salaries • Alumni perceptions of satisfaction and career responsibilities • Student ratings of knowledge and skills • Honors, awards, and scholarships earned by students and alumni • Student/alumni satisfaction with their learning

34. More Details on Student Learning Outcomes • describe specific behaviors that a student of your program should demonstrate after completing the program • focus on the intended abilities, knowledge, values, and attitudes of the student after completion of the program • What is expected from a graduate of the program? • What is expected as the student progresses through the program?

35. Why Are Student Learning Outcomes So Important? • basis for program improvement • instruction, course design, curricular design • communicate instructional intent • included in the syllabus • increase awareness of learning (for students) • common language • advising materials • promotional materials • support accreditation and evaluation

36. When Defining Student Learning Outcomes • frame the learning outcome in terms of the program rather than individual courses or individual students • each student will receive a B or better in the XYZ course • graduates from the program will demonstrate knowledge of engineering fundamentals • use concrete action verbs (e.g., define, classify, operate, formulate) rather than passive verbs (e.g., be exposed to) or vague verbs (e.g., understand, know) • action verb lists based on Bloom’s Taxonomy

37. Types and Levels of Student Learning Outcomes (Bloom’s Taxonomy) • cognitive: recall and intellectual skills • knowledge, comprehension, application, analysis, synthesis, evaluation • affective: attitudes, values, interests, appreciation and feelings toward people, ideas, places, and objects • receiving, responding, valuing, organization, characterization by, value • skills - perception, set, guided response, mechanism, complex overt response, adaptation, origination

38. When Defining Student Learning Outcomes (SLO) • do not to join elements in one statement that can not be assessed by single method • BSHE graduates will demonstrate knowledge of math, science, and engineering fundamentals, and gain competency in conducting oral presentations. • SLO statements should focus on the learning results and not on the learning process • computer applications will be introduced in all core engineering courses • BSHE graduates will demonstrate proficiency in XXX computer applications

39. When Defining Student Learning Outcomes (SLO) • SLO statements should be stated so that the outcome can be measured by more than one assessment method • students completing the XYZ engineering program will score over 95% on a locally-developed exam that tests application of engineering principles • students completing the XYZ engineering program will demonstrate competence and the ability to apply engineering principles • SLO statements should indicate the level and type of competence that is required of graduates of a program

40. Checklist for Student Learning Outcomes • aligned to mission and goal statements • clearly describe and define expected abilities, knowledge, values, and attitudes of the graduates of the program • simply stated • distinctive and specific to program • stated so that a single method can be used to measure the outcome • stated so that more than one measurement method can be used • focus on the learning results and not the learning process • measurable and there are resources available • can be used to identify areas to improve

41. Describing Measurement Methods • What are you going to use? • presentation, assignment, test, survey, observation, performance rating • Of and/or by whom? • student, mentor, focus group, customer, process, course • Context (e.g., where or when) • point-of-service, capstone, throughout the year, end of program • For what purpose • desired objective or learning outcome • Example: Test the students at the end of the program for their level ofknowledge in XYZ

42. Curriculum or Course-based • performance-based • capstone courses • capstone projects • case studies • classroom assessment • course-embedded assignments • course-embedded exam questions • portfolios • other • curriculum and syllabus analysis • content analysis of courses • reflective essays

43. After Identifying the Potential List of Measures You Need to… • select the “best” ones • try to identify at least two measure per objective or outcome • identify performance targets • balance between stretch targets versus achievable targets • example • survey (using the Graduating Senior Survey) the students at the end of the program for their level of satisfaction with their communication skills (indirect method) • 90% or more of the students will rate their level of satisfaction with their communication skills as “very good” to “excellent” on a survey (using the Graduating Senior Survey) that the students complete at the end of the program

44. Selecting the “Best” Assessment Methods • relationship to assessment—provide you with the information you need • reliability—yields consistent responses over time • validity—appropriate for what you want to measure • timeliness and cost—preparation, response, and analysis time; opportunity and tangible costs • motivation—provides value to student or customer, respondents are motivated to participate • other • results easy to understand and interpret • changes in results can be attributed to changes in the program

45. Challenges and Pitfalls • one size does not fit all—some methods work well for one program but not others • don’t try to do the perfect assessment all at once—take a continuous improvement approach • allow for ongoing feedback • match the assessment method to the objective and not vice-versa

46. After Identifying the Potential List of Measures You Need to… • develop assessment instruments • surveys • exams • assignments • scoring rubrics • portfolios • ideally you want them to be reliable, valid, and cheap • approaches • use external sources • seek help from internal sources (e.g., Institutional Research Office) • do it your self • the instrument may need to be modified based on assessment results