Good Rubric, Bad Rubric What ’ s the Difference?

Good Rubric, Bad RubricWhat’s the Difference? Kathleen Norris, MFA, EdD Plymouth State University

Good, or Emerging? Rubrics are a phenomenon of the profession of education. They have been embraced by some educators, tolerated by others, and loathed by some. Some educators are afraid of them, or don’t understand them. In many ways, rubrics are just like dogs…

Which ones are good?

Good Dogs • Do what you want them to do. • Are pleasing to look at, are not too big or too small for their breed or their bones; they fit their frame. • Are connected to you, are reliable. • Everybody experiences them as good dogs; there is agreement on their “goodness.” • No matter how good a dog might be, a person who’s afraid of dogs is afraid of all dogs and will stay as far away as possible.

Good Dogs

Good Rubrics • Do what you want them to do, assess what you want to assess. • Are pleasing to look at, are easy on the eyes, are clear to everyone, are not too big or small for their purpose. • Are reliable, valid, fair, and completely connected to what you are assessing. • Everybody understands the same thing when they read one, but they can still be scary.

Emerging Dog

Emerging Dog • Does things that he wants to do, not necessarily what I want him to do; might wander off on his own. • Could be in a bit better shape, is not quite the right weight for his frame. • Is connected sometimes, but is not completely reliable. • May seem “good” to some people but not so good, or even scary, to others, especially when he jumps up without warning.

Emerging Rubrics • Assess something, but not necessarily what you want them to assess. • Could be clearer, easier on the eyes; might need to be smaller or larger. • Are not completely connected to the standards, the performance (task), or both, so are not reliable, valid, or fair. • Not everyone understands them the same way and they can be very scary.

Student Voices; What’s Worst • "I'm kind of getting tired of rubrics because we even have a rubric at lunch which is rediculess [sic].” • "Some rubrics don't fully cover all the situations.” • "Most of the time it's the language. It can be hard to understand, especially the CAS2 (district test) rubric.” Reforming Middle Schools and School Systems: What students say about rubrics. http://www.middleweb.com/CSLB2CubRub.html

Authentic Assessment According to Grant Wiggins, “Assessment is authentic when we directly examine student performance on worthy intellectual tasks.”

Traditional Assessment Again from Grant Wiggins: “Traditional assessment…relies on indirect or proxy 'items'--efficient, simplistic substitutes from which we think valid inferences can be made about the student's performance….”

Authentic Assessment “A move toward more authentic tasks and outcomes thus improves teaching and learning: students have greater clarity about their obligations (and are asked to master more engaging tasks), and teachers can come to believe that assessment results are both meaningful and useful for improving instruction.” Wiggins, Grant (1990). The case for authentic assessment. Practical Assessment, Research & Evaluation, 2(2). Retrieved January 22, 2007 from http://PAREonline.net/getvn.asp?v=2&n=2.

Good Rubrics • Assess the performance of a student in the completion of a task. • Complement, but do not completely replace, traditional assessments. • Communicate expectations clearly. • Communicate the level of success a student is having. • Inform future instruction.

Rubric Development • Questions that need to be answered, and should be answered collaboratively: • What should the students know and be able to do? (These are the standards.) • What products indicate that students have met these standards? (Describe the authentic task(s) or project(s) that will be scored.)

Rubric Development • What does “perfect” performance on these tasks, or a “perfect” project look like, and what does an unsuccessful project look like? (These are the top and lowest score criteria for judgment.) • To communicate the expectations for the task, and, in the end, how well students performed, a rubric is developed that contains • the elements of the task or project that meet the established standards, • the criteria for the evaluation of each element, and • ultimately the student’s “score” on each element.

Good Rubric Development • Collaboration is essential in this work. • Define the standards to be met. • Describe the task(s) or project(s) to be completed. • Describe the criteria for judging the task or project on each required element. • Set the scale from “perfect” to unsuccessful on each required element.

Sample Writing Rubric

Designing the Rubric • Creating a table with scores across the top and criteria down the side and language within each box is easy, but you might be creating an “emerging dog” instead of your desired, and good, “dog.” • The language in each of the boxes is the key to a “good rubric.” • Rubrics that are “emerging” can be rehabilitated and become “good” ones!

Language Within a Rubric • Each “box” will have a description of what that score value means relative to the criteria. • The descriptions will be written in language that is appropriate for the grade level of the students and that avoids any jargon or language that would be unclear to parents.

Language Within a Rubric • The language will be agreed upon by the developers of the rubric, at a minimum. Best practice would be to have at least a sample of the students and parents involved in revising a draft of the rubric to raise any questions of clarity about standards, criteria or the task, and to assist in revising the language before it becomes the “final” draft.

Testing a Rubric • The “final draft” of a new rubric should be “tested” by having a sample of tasks provided to readers for scoring on the rubric. • Ideally, teachers, students and a sample of parents would be involved in this, but at a minimum the teachers would have collaborated on this rubric.

Validity and Reliability • The rubric can be considered valid if the different readers assign the same score values for each sample, • and it can be considered reliable if a set of “perfect” projects are all given “perfect” scores by the various readers, and a set of unacceptable projects also get unacceptable scores by the readers, etc.

Fairness • The task, or project, assigned must be designed as an authentic assessment of how well students have achieved the standards. • The students should be given the rubric with the assignment, and should be taught how to use the rubric to create their best work.

Fairness • The rubric must be applied equally to all tasks or projects; the identity of the student should not affect the scoring of the project. • The connection of the rubric to a final “grade” on the assignment must be made clear from the beginning.

FAQ’s • Choose to run scores from left to right, or right to left? How big should the matrix be? • We naturally read English documents from left to right. Some suggest starting on the left with the “perfect” score column so students see that first, but this can feel awkward to traditional learners. • The matrix should contain no more than 6-7 score points and no fewer than 3. Five seems to be enough for clarity for most rubrics. You can put +/- signs in between the columns if you need finer discrimination in scoring. • The number of criteria should be the minimum necessary to assess the project or task.

FAQ’s • Can a good rubric be created by a person all on their own? What about all the “rubric maker” software programs? • The best rubrics are developed and tested collaboratively. • If there are predefined standards and established criteria, an individual can create a good rubric. There are many free online rubric creating programs as well as websites and books with sample rubrics that can be adjusted. See the full page list of these websites on the handout. • The best rubrics are tested for validity and reliability and this requires collaboration. Authentic assessment in general will be used more effectively to inform instruction when teachers collaborate on the testing of rubrics.

FAQ’s • If you don’t have a collection of samples to use to test the rubric, how can you be sure that it’s reliable and valid? • You really can’t be sure. • If you have a “new” task or project then the rubric is being “tested” with it as the students complete the project or task. The “untested” rubric will have to be handed out with the assignment, but even at that stage it can be improved based on student feedback. • The particular challenge with reliability and validity comes with getting inter-rater agreement on the scoring and you need to have samples and more than one reader to establish the “goodness” of the rubric.

FAQ’s • Should the points given “add up” to a grade, or be averaged for a grade? • This really depends on the task or project and what the grade is meant to represent. If the task itself is getting a grade, then the rubric can be scored to produce a grade. However, many tasks are part of a larger graded unit, so there might be a rubric created for the group of assignments that all work together to produce a grade. • A writing assignment, for example, can be scored on a rubric for the quality of the writing, but there may be a number of elements to the written piece that must be included in order for the paper to be successful at meeting content standards that are outside of actual writing standards.

Rubrics • They are an integral part of the authentic assessment of worthy intellectual tasks and projects. • They can be used effectively to communicate with students and parents about what’s expected and how well a student is meeting those expectations. • They can inform instruction more effectively than traditional assessments.

Good Rubrics • They can be time consuming to produce, but valid, reliable and fair rubrics will last for as long as the standards are assessed by a task or project. The criteria are based on the standards, so they can be adjusted, if necessary, for different authentic assignments, without being rewritten. • They need to be introduced systematically for the educators, students and parents who may be “afraid” of them. Don’t let them “jump up.”

Student Voices; What’s Best • "In the end they help make things fair so that a student can't complain about his grade, and it also gives the teacher back up for the grade he/she gave. A student can also use it for them if they feel they have been unfairly graded.” • "I think rubrics can help teachers grade better than just thinking of a grade in their heads.” • "I like the way they word the scores and also how they show you to get that score. The guidelines are right down to the point and are precise to what you need to get that score. Transforming Middle Schools and School Systems: What students say about rubrics.http://www.middleweb.com/CSLB2CubRub.html

Task • Respond to this prompt on Edmodo: Select a rubric you have used in the past. Examine the rubric in light of the PowerPoint. Is it a good rubric or a bad one? Explain your answer. How could you make it better? • During the next professional development create a new rubric and share on Edmodo.

Good Rubric, Bad Rubric What ’ s the Difference?