Preparing Faculty for program and Learning Outcomes Assessment

1. Preparing Faculty for program and Learning Outcomes Assessment Community College Planning and Research Organization Terri Manning and Denise Wells Central Piedmont Community College

2. My Opinion (or bias) About Assessment • Learning outcomes assessment is a means to an end – not the end in itself. • It should exist within an overall evaluative process at colleges/universities where assessment is done for improvement of programs and services. • There is no such thing as a “one size fits all” model where we can all measure the same thing. • Institutions must have the freedom to determine their own outcomes based on their mission and the student body they serve.

3. Opinion, cont. • We must do a good job of establishing appropriate levels of effectiveness or government agencies will do it for us. We don’t need another IPEDS type of structure for learning outcomes. • The focus should be on outcomes, not just learning outcomes. All units and staff impact learning and student success therefore all units should define, measure and use outcome data to improve services and success.

4. The Problem with Continuous Meaningful Assessment • Institutions get the idea they are doing it for their accrediting agencies which creates the wrong foundation • Faculty and staff don’t respond well to time-consuming processes being forced on them and being told “SACS says we have to do this.” • The accrediting agencies are requiring it because good institutions do assessment and act/react appropriately. • This is not an end into itself. • It is part of a process leading to a much greater end.

5. Why Continuation is Hard • Often it is driven from the top down – administration tries to tell the faculty what needs to be done. • Only have support in small pockets. • It becomes one more thing we place on the backs of the faculty. • Keep adding more to their plates and taking nothing away. • My philosophy is “do as little assessment as you can and get meaningful, useful results.” • The “if 2 is good then 16 must be better” approach must be tossed out.

6. The Trick: Getting Faculty to Commit to Assessment • Assessment tells you what is going on with your students • What works and what doesn’t work • What skills are fully developed, improving or absent • Makes sure students have the critical skills • Makes sure students have the building blocks from which to improve

7. Good Assessment • Is meaningful to faculty and students • Should be authentic • Should be imbedded in the classroom • Becomes a critical tool for faculty • Informs good teaching

8. What is a Learning Outcome? • Learning Outcomes: • What changes in knowledge, skills, attitude, awareness, condition, position (etc.) occur as a result of the learning that takes place in the classroom. These are direct benefits to students. • Examples: general learning skills (e.g. improved writing and speaking abilities), ability to apply learning to the work environment (e.g. demonstrate skills in co-op), program-specific skills developed or enhanced (e.g. take blood pressure.)

9. What is a Program Outcome? • Program Outcomes: • The benefits that results from the completion of an entire program or series of courses. Are there benefits for students who get the entire degree versus those who take a few courses? If so what are they? • Typical examples are: licensure pass rates, employment rates, acceptance into 4-year schools or graduate programs, lifelong learning issues, contributions to society, the profession, etc.)

10. What is Assessment? • Assessment – the ability to appraise or estimate the attributes of a person, group or program. Tools of assessment can include checklists, inventories, observations, needs assessments, rating scales and all types of tests. • Measurement – the process used to assign numerals to objects or constructs according to rules so that the numbers have quantitative meaning. • Tests – tools used in measurement of knowledge, attitude and behavior. Tests and testing are specific to a defined circumstances, a period of time or set of outcomes.

11. Direct and Indirect Measures • Direct assessment involves looking at actual samples of student work produced in our programs. • Indirect assessment is gathering information through means other than looking at actual samples of student work, such as surveys, exit interviews, and focus groups.

12. What is Student Learning Outcomes Assessment? • “The assessment of student learning can be defined as the systematic collection of information about student learning, using the time, knowledge, expertise, and resources available, in order to inform decisions about how to improve learning. (p.2)” Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

13. Resource • National Institute for Learning Outcomes Assessment • http://www.learningoutcomeassessment.org/ • New Publication • Opening Doors to Faculty Involvement in Assessment • Author: Pat Hutchings

14. Obstacles to Faculty Engagement • The language of assessment as been less than welcoming. The language (accounting, testing, evaluation, measurement, benchmarking, etc.) come from business and education – not necessarily respected disciplines on campus.

15. Obstacles, cont. • Faculty are not trained in assessment. More focus has been placed on scholarly expertise in one’s field in doctoral programs with less attention paid to reflecting on educational purposes, formulating learning goals, developing assessments and exams and using data for improvement. Not enough professional development training offered. When the conversation turns technical, faculty bow out not wanting to appear amateurs. When colleges establish an assessment office with specialized staff to manage it, it can marginalize regular faculty.

16. Obstacles, cont. • The work of assessment is not a match with internal reward systems (promotion, tenure and merit). It is more valued in an environment where teaching is the first and foremost activity. In universities where research is valued more than teaching, assessment is undervalued and not part of the promotion and tenure conversation.

17. Obstacles, cont. • Faculty have not seen evidence that assessment makes a difference. Based on the demands on faculty time and energy – faculty must make a choice – and the rational choice is often “no assessment.” Institutions have a responsibility to use assessment results to improve the educational experiences of their students. The simple presence of data does not lead to improved outcomes.

18. A few other issues…. • In the early days the players in assessment were primarily outside of academe which made it look to faculty like “someone else’s agenda.” When faculty get the idea that this is not their job – you have a problem. •  Much of what has been done with assessment has not involved large numbers of faculty in a significant way. • Faculty need to be significant participants in the assessment process – not just token members of a committee pulled together for an accreditation visit or an after-the-fact audience for assessment results they had no part in shaping.

19. A few other issues…… • Faculty must ask (both individually and collectively) about their students: • What purposes and goals are most important for students? • Are these goals being met and what methods can we use to determine this? • How can we shape and share feedback that can strengthen student learning?

20. Resource • Assessment Clear and Simple: A Practical Guide for Institutions, Departments, and Genera Education • Jossey-Bass Publishers • Authors: Barbara Walvoord and Trudy Banta • $26.51 on Amazon.com

21. Assessment Characteristics • It is intended to: • to inform action • not to provide “proof” (too many variable) • provide the best available indicators about the attainment of department learning goals • include a variety of tools and not require standardized test or objective measures. • provide evidence on which to base decisions about curriculum, pedagogy, staffing, advising and student support on the best possible data about student learning and the factors that affect it.

22. Assessment Characteristics • Faculty regularly: • assess complex work in their fields • make informed professional judgments about critical thinking, scientific reasoning, or other qualities in student work • use those judgments to inform departmental and institutional decisions (p. 2) Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

23. Assessment Characteristics • A lot of assessment is already going on in responsible classrooms, departments, and institutions, though we have not always called it that. • We are asking faculty to take something they casually do and make it formal. • Assessment can move beyond the classroom to become program assessment

24. Assessment Characteristics • Classroom assessment – faculty evaluates her own students’ assignments in the capstone course and uses the information to improve her own teaching the next semester • Program assessment – faculty evaluates her own students’ assignments in the capstone course, outlining the strengths and weaknesses of the students’ work in relationship to departmental learning goals. The department uses the data to inform decisions about curriculum and other factors that affect student learning. Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

25. Methods of Assessment • Course imbedded assessments for learning outcomes which can be (direct): • Written works • Student journals • Speeches • Skills-based assessment (demonstrated skills) • Observation checklists • Tests for specific skills • Teamwork assessments

26. Methods of Assessment • Indirect Assessment Methods • Surveys that ask about specific behaviors indicative of changes in values and attitudes • Not self-evaluation of perceptions of learning • Surveys that ask about effectiveness and efficiency • Qualitative methods • Focus groups, key informant interviews, open ended surveys, journaling • Portfolio analysis

27. Authentic Assessment • A form of assessment in which students are asked to perform real-world tasks that demonstrate meaningful application of essential knowledge and skills. An authentic assessment usually includes a task for students to perform and a rubric by which their performance on the task will be evaluated.www.oaklandcc.edu/assessment/terminology.htm

28. Authentic Assessment Examples • Authentic assessment tends to focus on complex tasks, enabling students to demonstrate their competency in a more 'authentic' setting. Examples of authentic assessments include: • performance of the skills, or demonstrating use of a particular knowledge • simulations and role plays • studio portfolios, strategically selecting items • exhibitions and displays

29. Toughest Concept • Difference between grading and assessing learning outcomes.

30. Grades vs. Outcome Assessment • Course Grades are a measure of the percentage of all the requirements for a course that a student completes. • Course requirements often include things like attendance, class participation, turning work in on time, following the rules on an assignment, etc.

31. Grades vs. Outcome Assessment • It is possible for a student to make an A and have learned little or nothing. It is possible for a student to earn an F and have learned more than anyone in the class. • Grades are - "An inadequate report of an inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite material.“ Ohmer Milton, Howard Pollio and James Eison, Making Sense of College Grades, (Jossey-Bass, 1986).

32. Measurements of Learning Outcomes • Poor example • Student writing skills will improve • 70% of students will pass the midterm - or • 70% of students will pass the final – or • 70% of students will make A grade of C or better in ENG 111 • Good example • Students will demonstrate the ability to locate, critically evaluate, and present information • 70% of student persuasive speeches evaluated will receive at least a score of 3 or better on a 5-point evaluation rubric designed to measure effective speaking along a continuum on six construct scales.

33. Methods of Assessment • Surveys that ask about specific behaviors indicative of changes in values and attitudes • Not self-evaluation of perceptions of learning • Surveys that ask about effectiveness and efficiency • Qualitative methods • Focus groups, key informant interviews, open ended surveys, journaling • Portfolio analysis

34. Assessments of Behavior Changes • In what behaviors do we observe changes in students? • Teamwork • Cultural awareness • Fear of pubic speaking • Ethic issues • Math phobias • How might we measure these?

35. Assessment • Is a developmental, iterative process • Like testing swimming pool water

36. What We Do With Our Faculty A Seven Step Process

37. First - Orientation • Instructional administration attends to show support • Review process • Review of timelines • Preview of program review • Introduce players and their skills • Have added marketing and recruitment

38. Second: Provide Training • Nuts and Bolts of Outcome Assessment • Defines types of outcomes • Establishes reasons for assessment • Identifies sources for outcomes • Give examples • How to Read Your Data • Faculty need to see their student success data (grades, retention, graduation, etc.) but they are not necessarily good with numbers • The assessment just establishes that students are mastering the learning and program outcomes so they can move forward to the items above

39. Seven Step Assessment Process Lots of hand-holding and assistance • Discovery - During this phase, faculty members are involved in a working session where they are trained on learning outcomes assessment. They move into an activity on identifying “reasons for being, what we want them to get out of it and overall benefits for students.” Outcome Identification Form • Prioritization - During this phase they begin the narrowing of a long list of potential outcomes to the critical ones. This process involves them working through 1) what would be useful for faculty and departments to know, 2) what the faculty/department values, 3) what would inform decision-making and 4) what is most critical for student success. Outcome Prioritization Form

40. Seven Steps (cont.) • Operationalization - During this phase they create operational definitions of each outcome based on the curriculum course or program (e.g. effective writing is defined differently for English faculty than it is for early childhood faculty). Outcome Statements Form • Mapping - During this phase, faculty look at the natural fit between 1) courses and outcomes and 2) assessments/projects and outcomes. They create assessment processes where good ones are lacking. Outcome Assessment Form

41. Seven Steps (cont.) • Measurement - This phase involves setting realistic outcome targets, developing/using assessment tools, the process of distribution and collection of data and the reporting chain of command. • Analysis and Use - This phase involves analysis among appropriate faculty, determining a plan of action and writing up results. • Follow-up one year later on action plans (strongly recommended.)

42. What’s Needed to Get the Job Done? • Training • Time Saving Short-cuts • Templates and processes • Website support and assistance • Survey/assessment assistance • Coaching and mentoring from IE • Getting faculty buy-in

43. Bring Faculty On-board • Pat Hutchings suggests six ways to bring the purposes of assessment and the regular work of faculty closer together in the publication: Opening Doors to Faculty Involvement in Assessment

44. 1. Build assessment around the regular, ongoing work of teaching and learning • Build on the process of grading in every course every semester. This can bring forth questions about course design, assignments and exams and feedback to students. This is where faculty talents and interests lie. Embedding assessment in the classroom sets the stage for discussions at the program level and draws on what faculty care most about: their discipline or field. When assessment reflects and respects disciplinary interest, it is more likely to lead to important faculty engagement.

45. 2. Integrate assessment into the preparation of graduate students; • Finding this is the exception rather than the rule but it is occurring more often.

46. 3. Reframe assessment as scholarship • Student learning should be seen as an important phenomena for investigators. The work should not be seen as service to the college (serving on the assessment committee) but as an important intellectual exercise. Attention should be paid to the use of new forms formulas, and genres for capturing the scholarly work of teaching, learning and assessment.

47. 4. Make a place for assessment in faculty development •  Many institutions of higher education have created centers for teaching and learning and assessment can be part of these centers. Bringing faculty together and facilitating constructive conversations around assessments, meaning, implications for pedagogy and a commitment to evidence can increase faculty engagement.

48. 5. Create campus spaces & occasions for constructive conversations and action • Teaching and learning have been traditionally seen and undertaken as private activities. Faculty assume responsibility for their own students and classrooms. It is foreign for them to come together in groups and discuss student learning, difficulties, strategies and possible actions as groups. Institutions have done various things from faculty learning communities, setting aside time in department meetings, multidisciplinary reading groups and inquiry groups to foster these conversations.

49. 6. Involve students in assessment. • Student self-assessment where they monitor and direct their own development is gaining groups. New products exist such as e-portfolios, rubrics to guide student work and involve students in campus conversations about learning and assessment.

50. Working through the process… • Mentoring • Reviewing data, reviewing what research is, inputting and formatting data • Calendars • Reviewing deadlines • Putting Learning and Program outcomes into action • Using data for results