The Role of Calibration in Advancing Faculty Learning About Student Learning

1. “Our” Common Ground: The Role of Calibration in Advancing Faculty Learning About Student Learning Terry Rhodes & Ashley Finley AAC&U Institute on Integrative Learning and the Departments Portland, OR

3. The Anatomy of a VALUE Rubric Criteria Levels Performance Descriptors

4. The Calibration Training Process • Scoring Steps: • Review rubric to familiarize yourself with structure, language, performance levels • Ask questions about the rubric for clarification or to get input from others regarding interpretation • Read student work sample • Connect specific points of evidence in work sample with each criterion at the appropriate performance level (if applicable) • Calibration Steps: • Review scores • Determine common score(s) • Hear from outliers • Discuss • Determine final score

5. The Ground Rules • This is not grading. • Think globally about student work and about the learning skill. Think beyond specific disciplinary lenses or content. • We are not changing the rubric (today). • Our work is time sensitive. Go with your instinct. • Start with 4 and work backwards. Pick one performance benchmark per criterion. Avoid “.5”. • Zero does exist. Assign “0” if work does not meet benchmark (cell one) performance level. N/A exists. Assign “not applicable” if the student work is not intended to meet a particular criterion.

6. Signature Assignments • Assignment should enable attainment of criteria • Break down criteria to determine key components for assignment • What should students do with content to meet criteria? • E.g. What are the pieces to be analyzed, compared, integrated? • Will the assignment be used for more than one outcome? • What are the types of assignments that will be most helpful for allowing students to demonstrate competency?

7. Step 1: All Gen Ed Courses reported as addressing and assessing Info. Tech. Literacy identified as potential courses from which to request artifacts. (54 courses) Step 2: Of courses identified, approx. 20% were randomly selected for sample (10 courses, 36 total sections) Step 3: Within each selected course, 2students randomly selected by roster # to submit artifacts (74 artifacts) Step 4: Start of semester, department chairs notified of courses in from which artifacts were to be requested. Chairs worked with individual faculty to fulfill request. Step 5: Artifacts submitted to Director of Learning Outcomes for scoring. (66 artifacts) Step 6:Faculty scoring team met at the close of spring semester for a norming session and scoring.(62 artifacts) Example of Process From: Carroll Community College Flow chart of sequential steps in the request, submission, and scoring of student artifacts for Learning Goal 4: Information and technology literacy.

8. Campus Example of Outcomes Assessment Using Rubric data From: UNC-Wilmington, Critical Thinking Rubric

9. Building the Evidentiary Base: University of Kansas Critical Thinking: Issues, Analysis, and Conclusions Inter-rater reliability = >.8 Percent of Ratings

10. Building the Evidentiary Base: University of Kansas Critical Thinking: Evaluation of Sources and Evidence Percent of Ratings

11. Building the Evidentiary Base: University of Kansas “VALUE added” for 4 years - writing Percent of Ratings