1. Designing Rubrics For Classroom Assessment Professor Timothy Farnsworth, CUNY Hunter College

2. Communicative Assessment Is Hard! • We are making inferences about what people can and can’t get done in the real world, not: • What grammar or vocabulary they can choose from 4 options on a MC test • Tasks must be realistic but can’t be TOO realistic: • Time constraints • Stakes are different • Interlocutors are different • Setting, format usually different

3. Speaking Assessment Challenges • Speaking is the hardest skill to assess • Impermanent -> must be rated on the spot or recorded • Interactive -> Real communication takes two persons • Nonverbal -> Much real communication takes place without language use • Topical -> Students’ knowledge base affects their speaking output

4. Oral Assessment / Testing Tasks • Must get students involved in speaking! • In communication, many things happening • Listening • Nonverbal channeling • Decisions made about what to say, what not to say • Students practice avoidance strategies, game the system, don’t take risks • Assessment tasks take many forms

5. Types of Assessment Tasks • Many options – some common ones: • Interview, either scripted or unscripted • Picture description • Role Play • Pair or group tasks • Almost any classroom task, really!

6. Sample task: Pair oral assessment • Each student has a different picture; communicate to find similarities and differences

7. Scoring tasks and communicating results • We need some way to turn the “observation” into a “score” or “grade” • Usually done via numbers, can be done as letters, smiley faces, etc. – doesn’t matter • Should be clear, simple • Should be “reliable” -> repeatable, dependable • Should be “valid” -> meaningful • Shouldn’t cause any harm

8. Types of scoring rubric • Holistic • Analytic • Task Fulfillment (“primary trait”) • Performance decision trees • Checklists

9. Holistic Rubrics • One overall score tied to a “performance descriptor” • Most familiar type of rubric • Assign score of the description that best matches performance • Problem: Performance doesn’t match description at the same level in all areas • Advantage: Simple to use and understand

10. Analytic Rubrics • Two or more categories of scores • Usually based around linguistic categories such as pronunciation, grammar, etc. • Also quite common • Assign scores for each category per performance • Problem: Can be cognitively challenging for raters • Advantage: provides more detailed information

11. Task Fulfillment Rubrics • Usually look like holistic rubrics • Instead of describing language, they describe whether the student completed the “task” • Usually used in role play type assessments • Useful in testing for employment, other settings • Useful when there is a specific set of tasks that involve language and can be specified • Example rubric for hotel employee testing: • Job applicant is tested for ability to respond to guest requests and complaints, answer telephone, etc. in role play situation,

12. Performance Decision Trees • Fairly new to the scene(Fulcher 2011) • Use a flowchart design • Good for clear, simple scoring when the task is very consistent / clear • Give points for each successful “yes” on the flowchart • Lets the test raters answer very specific questions

14. Self / Peer Assessment • Students have some good ideas about their and their classmate’s performances • In a busy classroom, teacher cannot hear everyone • Students can use any feedback they get • Training students to listen carefully and help each other is good practice in general • But: Students are not always good evaluators of their peers • May not catch pronunciation or other important mistakes • May be LESS proficient than the person they are supposed to assess • Only ask them to assess in areas they can definitely “do” – yes / no checklists are great for this

15. Checklist Rubrics • Phrased very simply like a can-do statement: • “Student can understand and use vocabulary for classroom objects. Yes / No / Partially • “Student can use formal language in giving oral presentations” Yes / No / Partially • Can be broken into parts: • “Student understands WH questions” Yes / No / Partially • “Student responds to WH questions using appropriate grammar and vocabulary” Yes / No / Partially • “Student responds to WH questions at length and returns questions appropriately” Yes / No / Partially

16. Checklists for student self-evaluation • Self Evaluation – teacher collects evidence • Checklists are simple and clear (example) • List main objectives for the week, create checklist for students • Have them do and turn in anonymously • Use the info to plan review sesssions in future

17. Keys to consistent, fair scoring 1: KISS principle for tasks • Keep It Simple, Sweetheart! • Don’t use any more complicated assessment than you really need • One or two tasks is enough as long as you are getting the right kind of interaction • Make sure you get “enough speech” to make a rating using the right rubric • Make a system that everyone can easily understand • Good speaking testing usually can be done in less than five or ten minutes • More “high stakes” testing takes longer, up to an hour

18. Keys to consistent, fair scoring 2: Use the simplest rubric that works • I like checklists or performance trees for classroom and self assessments • I like analytic rubrics for more high-stakes testing • They give information on what things the student needs to work on • Holistic rubrics don’t provide much information for students • I don’t like task fulfillment rubrics for most situations • It’s not clear from the score what the student needs to work on • It’s sometimes possible to do a task using little language; the task fulfillment setup still allows high scores

19. Keys to consistent, fair scoring 3: Write the rubric so students can understand it • Use plain and simple language • Use “I can / can’t do X” or “You did X” when possible • Explain the scores and the rubric to them before the test – • no reason for it to be a secret • They should understand why they got their score and what they can work on next time • Do not use a self assessment for a grade • Do not use peer assessment for a grade

20. Your assignment: Write a scoring rubric • Find a partner next to you • Together, pick one rubric style: Holistic, analytic, decision tree, checklist, task fulfillment • Think about students in a typical class you teach: • What level are they? • What do you need to learn more about in terms of their speaking skills? • Write a rubric with a few levels and clear descriptions • Compare your rubric with a pair near you – does it look similar? Why or why not?

21. Sample task: Pair oral assessment • Each student has a different picture; communicate to find similarities and differences