Designing Assessment Materials

1. Designing Assessment Materials Delsea School District November 2-3-4, 2009 Bob Shamy: Director of Professional Development Services

2. How well does it match the content/skill? Are you assessing significant content/skills? Did you clarify the expectations and focus on the content/skill being assessed. Will the students find the test interesting? High Stakes Testing Sensitivity/Bias? Reliability Does it address the enduring understanding and essential questions? Developing Assessments

3. Traditional Forced Choice Body of Knowledge is Determined This Knowledge Becomes the Curriculum Assessments: Did Students “Get” the Curriculum Curriculum Drives Assessment Authentic What student tasks will allow them to demonstrate mastery of the content/skill? Curriculum is developed based on these tasks or assessments. Assessment Drives Curriculum/Learning Traditional and Authentic Assessment

4. Assessment • Assessment can help evaluate both the student’s progress and the teacher’s effectiveness. • Assessment measures whether or not the objective has been learned. • The question assessment tries to answer is simple: Did the student’s achieve the lesson’s objectives? • Assessment efforts that don’t answer that question miss the mark. Student Assessment Link

5. Formative Assessment Part of Instruction and Tied to the Objective Allows for Adjustments During the Learning Process for Both Teachers and Students Focuses on What Students are Learning and/or Have Just Been Introduced Practice Allows Teachers to Define the Components of Summative Assessments

6. ASSESSMENT FIRST What content or skill do you want students to gain by the end of this lesson/unit? Content/Skill Assessment Know Show Planning methods of assessment first: • Clarifiesthe development of your methods and activities • Focuses the teacher on the development of critical thinking questions and closure • Directlyconnects the objectives/enduring understandings to the assessment

7. Lesson Interaction/Assessment T Reinforce Objective/ Check for Understanding T Collective Review/ Check for Understanding Teacher Clarifies Objective Assessment of Objective Closure Small Group Or Independent Work Students Entire Class StudentsGuided Practice

8. Closure • Reviews the objective • Draws the learned material together • Assists students in internalizing the new learning • Prepares students for what is to come • Assesses student outcomes Exit Cards Think/Pair/Share Whip Around Pose a Question for the Next Lesson’s Do Now and Assign it for Homework The “CLIFFHANGAR” Sets up the New Lesson

9. Summative Assessment • ENDURING UNDERSTANDING • Important Questions that Occur Throughout Your Life. Arguable But Enduring: What is Justice? • ESSENTIAL QUESTIONS • Key Inquiries Within a Disciplines. Big Ideas that Relate to a Content Area or Subject: Is the American Judicial System Based on the Equitable Distribution of Justice? • What is Needed to Learn Content. Helps Students Make Sense of Big Ideas. What is the Significance and Symbolism Behind Lady Justice?

10. Steps to Be Used in Developing an Assessment • Decide on content and skill objectives or standards to be measured • Decide on the allocation of items and score points for each standard or objective • Decide on levels of cognition (knowledge, comprehension, etc.) to be emphasized • Decide on what item formats to use • Decide how much emphasis to give each format

11. What are their purposes? What forms may they take? Where may they be found? Are they appropriate? Are they free of biases and stereotypes? Is permission needed for using them? Are they being cited properly? Thinking About Stimulus Materials

12. Drafting Multiple Choice Items • Determine the item’s main idea • Construct the stem and key • Draft the 3-4 distracters (incorrect answers) • Review the item for significance • Refine items according to guidelines

13. Check for: One correct answer Keys in varied positions Clear direction provided by stem Parallel format of options Plausibility of options Clueing the key What Guidelines Should Be Used When Drafting Multiple-choice Items?

14. Multiple Choice Test Items There are two types of MC items in relation to answers: A. Correct Answer – Simple facts, one and only one right answer 1. Lincoln was born in A. Illinois B. Kentucky (correct answer) C. Ohio D. Indiana • Best Answer- Who, what, when and where; more complicated, requires understanding, application and interpretation of factual material. 2. Which of the following factors is most significant when choosing a state capitol? • a. Location (best answer) • b. Climate • c. Highways • d. Population Be sure that your correct answer can be defended based on the content of the lesson.

15. The stem of the item should present a definite problem: • Poor:South America • a. Is a flat, arid continent • b. Imports coffee from the United States • Has a large population • Was the destination of Columbus. • e. Was settled by colonists mainly from Spain (correct) • Better:Most of South America was settled by colonists from • a. England d. France • b. Holland e. Spain (correct) • c. Sweden

16. BEST:Most of South America was settled by colonists from Spain who sought to: • a. expand their empire and bring in more revenue. (Correct) • b. connect northern territory to South America. • c. create permanent colonies that would grow and prosper. • d. generate a thriving fur trade. • e. build relationships with indigenous people without force.

17. Poor: The Harlem Renaissance • a. involved many African American artists. • b. was a movement of mainly writers. • c. began with the work of Langston Hughes. • d. occurred in the 1920’s. • (This item is ambiguous. A and D equally correct.) • Better: What was the Harlem Renaissance? • a. A political movement. • b. An artistic movement of African Americans . (correct) • c. A school of landscape painters. • d. An architectural style. • Best: The Harlem Renaissance was a movement of African American writers, painters and musicians in the 1920’s in New York. Most were interested in • a. Making money • b. Creative self-expression • c. Travel • d. Living in the city

18. Use a negative stem only when necessary: • Poor: Which one of the following states is not located north of the Mason Dixon line? • a. Maine • b. New York • c. Virginia • Pennsylvania Better: Which of the following states are located south of the Mason-Dixon Line? • a. Maine b. New York • c. Virginia d. Pennsylvania Negative items tend to focus on unimportant learning: the least effective method, the least compelling aspect of something. In this item presumably we want the student to know which states are below the Mason Dixon Line.

19. Poor: Which one of the following is not a safe practice when driving on an icy road? • a. Accelerating slowly. • b. Jamming on the brakes. (Correct) • c. Holding the wheel firmly. • d. Gradually slowing down. • Better: All of the following are safe driving practices on icy roads EXCEPT: • a. Accelerating slowly. • b. Jamming on the brakes. (Correct) • c. Holding the wheel firmly. • d. Gradually slowing down.

20. Choices (distracters) should be grammatically parallel (consistent) with the stem. • Poor: An electric transformer can be used • a. For storing electricity • b. To increase the voltage of alternating current (correct) • c. It converts electrical energy into mechanical energy • d. Alternating current is changed to direct current. • Better: An electric transformer can be used to • a. Store up electricity • b. Increase the voltage of alternating current (correct) • c. Convert electrical energy into mechanical energy • d. Change alternating current to direct current

21. All distracters should be plausible. • Poor: Who discovered the North Pole? • a. Christopher Columbus • b. Ferdinand Magellan • c. Robert Peary (correct) • d. Marco Polo • Better: Who discovered the North Pole? • a. Roald Amundsen • b. Richard Byrd • c. Robert Peary (correct) • d. Robert Scott • While this falls within knowledge in Bloom’s Taxonomy all distracters are homogeneous, drawn from the same field of endeavor and time period. Students must know more about the historical period in order to choose the correct answer.

22. Avoid verbal associations between stem and correct answer. • Poor. Which one of the following agencies should you contact to find out about a tornado warning in your locality? • a. State farm bureau • b. Local radio station (correct) • c. US Post office • d. National office • (Use of the word local is a cue.) • Better. Which one of the following agencies should you contact to find out about a tornado warning in your locality? • a. Local farm bureau • b. Nearest radio station • c. Local post office • d. Local weather station (correct)

23. Use all of the above or none of the above sparingly. These are only useful with correct answer items. • Poor items: • 1. Which of the following is not an example of a mammal? • Bird (correct) • Dog • Whale • None of the above • (Distracter (d) is also not a mammal.) • 2. When the temperature drops, the pressure tends to • Increase • Decrease • Stay the same • None of the above • (There are no alternatives to choices a, b, or c.)

24. Used on High Stakes Tests Students Can’t Guess Their Way to a Right Answer To assess in-depth understanding To allow students to organize their answers To give students opportunities to be creative To encourage students to produce their own answers to problems To allow students to respond to multiple parts of an item To help students learn that it is important to be able to communicate clearly to others Constructed Response Items When to Use Constructed Response Items

25. Essentials when Developing Constructed-response Items • A question calling for a student response, sometimes preceded by a stimulus • Those parts of a scoring guide that identify • Criteria for a quality response • Standards which must be met at each score point level (RUBRIC)

26. It assesses one or more standards It assesses taught content and skills It assesses knowledge and skills, not beliefs, opinions, or values It provides direction and focus Students are familiar with the scoring guide’s values and criteria Its scoring guide relates directly to question It is independent from other items It will engage and interest students It is free from bias and stereotypes Evaluating a Constructed-response Item?

27. Focus on the proficiency you want to assess. What do you want students to demonstrate? What level of critical thinking are you aiming for in the question? Predict, Infer, Analyze, Criticize, etc. Check Bloom’s Taxonomy. Create a draft of your question. Check to make sure that you covered the content or proficiency. Formulate a response and/or ask a colleague to sketch out a response. Steps to Developing Constructed Response Items

28. Solid: The story "The Three Little Pigs" has many characters. Choose one character and describe two traits of the character. Explain how the character's traits lead him to act the way he does. it requires a high level of thinking clearly states what students should do and is answerable given the information in the story Solid versus not-so-solid CR Questions • Not-So-Solid: The story, "The Three Little Pigs" has many characters. Identify three characters in the story. Tell what each made their house out of. • recalls only trivial information • not all the characters in the story built houses, so the two parts of the question do not fit together. http://www.writingfix.com/RICA/constructed_response.htm

29. http://www.writingfix.com/RICA/constructed_response.htm

30. Developing a Constructed Response Scoring Guide • Working backward from top, determine what constitutes the various lower levels of performance for each criterion • Identify how many points to allocate for the item • Identify criteria for score-point responses • Allocate points to each criterion • Share scoring guide with students and specifically review criteria • Provide models of high, middle and low point scores. Examine Student Responses to Evaluate Scoring Guide