Designing High Level Open Response and Multiple Choice Items for Classroom Assessments

1. Designing High Level Open Response and Multiple Choice Items for Classroom Assessments KLA Regional Meetings August 2007

2. Definition • What is an open response question? • At your table, write a good definition for what you believe it to be. • You will have five minutes

3. Open Response QuestionDefined An Open Response Question (ORQ) is defined by KDE as a: question requiring you to think and apply what you have previously learned. You are asked to use your knowledge rather than just give facts. http://www.kde.state.ky.us/comm/commrel/taketest/act2.asp

4. Fair and Credible Assessments • Core content for assessment=classroom activities=assessment • Students must be assessed on what they have been taught • Content/activities/assessments must consistently reflect the same DOK level • CATS should not be a surprise but a continuation of on going assessment practices

5. 5 Types of ORQs • Single Dimension • Scaffolded • Two or More Parts • Student Choice • Respond to Information

6. Single Dimension • Asks a straight forward question which requires explanation • Example: As the United States became increasingly industrialized many aspects of American life improved. However, new problems also arose. • A) What are some problems that arose? • B) How was the growth of industry directly related to them?

7. Scaffolded Questions • Have multiple parts • Each question labeled separately • Each subsequent part relies on the student getting the first part correct • Example: In the story you have just read, Trevor is a complex character. • What traits would you use to describe him? • How do these traits affect the people around him?

8. Two or More Independent Components • Question has multiple parts • One part does NOT rely on the previous part being answered correctly • Example: This picture shows a sealed bottle with some water in it. It is a model of the water cycle. • Use what you see in the bottle and explain the water cycle. • Predict how life on earth would be different if water did not go through the water cycle.

9. Student Choice • Provides options for students to choose from • Example: You must teach your friend to strike a ball in one of the following activities: a. Golf b. Softball c. Tennis d. Baseball Select ONE activity and describe the proper steps in preparing, striking and ending a swing.

10. Response to Provided Information • Student responds to passages, data or graphics • Example: The students in Mr. Cupp’s class bought candy that came in small bags. Each student reported how many candies were in their bag. Here are the amounts: 16, 19, 18, 19, 20, 17, 21, 18, 19, 18, 16, 15, 14, 18 , 17 • Make a graph, table, chart or organized list that demonstrates the above data • Mrs. Cupp also bought a small bag of candy. Based on the data above, how many candies do you predict are in her bag? Explain your thinking.

11. Choosing Appropriate Type • Essential Information • Process/Skills • Answer look fors • Students should be assessed using ALL types of ORQs

12. Essential Information • Consider your essential question/information • Which question type allows the student to completely answer the desired content? • If using more than one, which combination will allow the student to demonstrate what they know and are able to do? • Is any piece missing?

13. Process Skills • What do you want them to be able to do? • Which type lends itself better to the actual skill you are assessing?

14. Answer Look Fors • Make sure the type of question will yield the complete answer you are expecting. • Students should be able to clearly and thoroughly demonstrate what they know and are able to do.

15. A GOOD OPEN RESPONSE QUESTION: • Can have many correct answers • Is more difficult to score • Takes longer to score • Allows students to demonstrate what they know and are able to do • Lots of work up front, but needs to be altered only if core content or teaching assignment changes

16. Question Development • Question must be able to be answered within a single page • Question must have a title • Scenario must be present • Must be core content related • Age Appropriate • Requires critical thinking • Four levels of responses are available

17. Question Development • Identify core content that was TAUGHT and assess only that content (backload) • Determine the type of question that is appropriate • Write question- remember format and HOTS • Develop a scoring rubric • Write model answers

18. Knowledge=Thinking • Curriculum concepts are to be learned deeply (DOK). • Teaching must engage ALL students in ACTIVE reasoning at all grade levels.

19. Classroom Rigor Includes: • Commitment to a knowledge core • High thinking demand • Active use of knowledge • Challenging, high level assignments • Student focused assessments at the mastery level (DOK) • Thinking is infused throughout the curriculum

20. Essay Vs. ORQ? • Many open response questions are actually essay questions in disguise. • Format alone does NOT make a question open response. • What is the critical thinking skill? • What are you asking them to do? • What are you looking for in the response?

21. Essay Vs. ORQ • It is an essay question if they are only supplying back information you provided to them through direct instruction, textbooks, or other instructional materials. • Remember definition of ORQ.

22. Essay Vs. ORQ • Decide at your tables which questions are essay type and which are ORQs.

23. Essay vs. Open Response • We have been studying the Civil War. Civil wars generally have multiple causes. A. Identify THREE causes of the Civil War. B. Explain how each contributed to the conflict. Essay or ORQ? ESSAY

24. Essay vs. Open Response 2) Think about the experiment we did in class using the balloon. A. Name TWO properties of air. B. Explain ONE of them using the balloon experiment. Essay or ORQ? ESSAY

25. Essay vs. Open Response 3) We have been studying Martin Luther King in class. A. List THREE important events in his life. B. Select ONE event and describe how you think this event helped him to advance race relations in America. Essay or ORQ? ORQ

26. Writing Higher Level Multiple Choice Questions

27. Writing Higher Level Multiple Choice Questions DOK LEVEL 1 • Recall of information, such as recalling a fact, definition, term. • Location of information (from a passage or graphic) • Students use simple skills and abilities to recall.

28. Writing Higher Level Multiple Choice Questions DOK LEVEL 2 • Processing of information beyond recall or reproduction. • Students must make some decisions as to how to approach the questions or problem. • Uses more than one mental or cognitive process / step.

29. Writing Higher Level Multiple Choice Questions DOK LEVEL 3 • Requires deep understanding. • Students must plan, use evidence and use more demanding cognitive reasoning. • The cognitive demands at Level 3 are complex and abstract.

30. Writing Higher Level Multiple Choice Questions DOK LEVEL 4 • Requires high cognitive demands and is very complex. • Students must make connections – relate ideas within the context or among content areas – and have to select one approach among many alternatives on how the situation can be solved. ** VERY difficult to write for multiple choice at this level.

31. DOK 2 This item is an application of computational algorithms. It is a multi-step problem requiring the student to make a decision of how to approach the computations.

32. DOK 1 The student must recognize or identify a rotation.

33. 1. What purpose does repeating the phrase, “and Brutus is an honorable man,” serve in Marc Antony’s first speech? A. to raise doubt about whether Brutus is an honorable man B. to emphasize that Marc Antony agrees with Brutus C. to express Marc Antony’s grief over Caesar’s death to praise Caesar’s life and accomplishments DOK 2 The answer requires processing of text; the answer isn’t directly stated in the text.

34. 11. The Sumerians chose 60 as the base for their number system because A. there are 60 minutes in an hour. B. they invented dial clocks. C. 60 can be divided easily. D. there were six sacred directions on the Sumerian compass. DOK 1 The question requires locating/recalling information.

35. 2. Which two persuasive techniques does Marc Antony use in his first speech? A. logical reasoning and name-calling B. emotional appeal and imitation C. circular reasoning and testimonial D. rhetorical questions and repetition DOK 3 Students aren’t just identifying a persuasive technique. They have to determine what is persuasive by finding examples and then have knowledge of the techniques. It involves reasoning.

36. Sources for Multiple Choice Samples Released Items from KDE’s website: • http://education.ky.gov/KDE/Instructional+Resources/Curriculum+Documents+and+Resources/Released+Test+Items/ Released Items Annotated to the DOK: • http://education.ky.gov/KDE/Instructional+Resources/Curriculum+Documents+and+Resources/Core+Content+for+Assessment/Released+Items+Annotated+to+the+DOK.htm

37. Steps in Designing a Multiple Choice Question • Decide which core content standard you wish to assess. • Identify the major learning target within the standard that you wish to assess. • Decide how deeply to assess the standard (DOK level). • Design the stem or question.

38. Designing the Question Stem • Match question to the standard • Use clear wording, keeping your focus on finding out what students know • Ask a full or complete question • Highlight (bold, CAPS) critical words • Have a colleague review it

39. Designing the Key / Distractors • Have ONE and ONLY ONE correct answer • Include only plausible answers • Use clear wording that is age / grade level appropriate • Maintain a consistent / parallel style and length • Logically order response items involving numbers or dates • Avoid using “all / none of the above” options • Have a colleague review them

41. New Resource from KDE for ORQ and MC Support:

42. Go to: http://education.ky.gov/users/jwyatt/PlanningGuide/Developing%20KCCT-like%20Questions.pdf