What Went Awry: the test questions, the curriculum, the instruction, or student comprehension?

1. What Went Awry: • the test questions, the curriculum, the instruction, or student comprehension? Presented by Charlie Adamovic, Jim Mshar, and Kevin Murphy January 9, 2012

2. Goals: • To Collect Data Using GradeCam • To Organize Data Using GradeCam • To Analyze Data

3. Vocabulary: GradeCam:A software program that, in combination with an optical reader, scores tests and allows for easy item analysis. Discrimination (discriminator): The effectiveness of an item to separate students who vary in their degree of knowledge of the material tested, and their ability to use it. For our purposes, we typically use the top 20% (high achievers) and lowest 20% (low achievers) of scores on each as discriminator set on EACH test.

4. Example of Instructional Inconsistency #11. Can kinetic energy ever be negative? Why? a. Yes, if the velocity is negative. b. Yes, if the work done is negative. c. No, because velocity squared and mass are never negative. d. No, because the object would travel backwards in time. Overall Data: 26/40 = 65 % Students answered correctly 14/20 = 70 % of High-Performers answered correctly 12/20 = 60 % of Low-Performers answered correctly

5. However… a Closer Look … Teacher #1 Data: 13/24 = 54 % Students answered correctly 7/12 = 58 % of High-Performers answered correctly 6/12 = 50 % of Low-Performers answered correctly Teacher #2 Data: 13/16 = 81 % Students answered correctly 7/8 = 88 % of High-Performers answered correctly 6/8 = 75 % of Low-Performers answered correctly

6. Clearly there was some difference between the instruction in Teacher 1’s classes vs. Teacher 2’s classes. This lends to an opportunity to share teaching strategies to help improve overall instruction.

7. Example of Good Discriminator #18. The net work done on the crate is ____. a. positive b. negative c. depends on what you assigned as the positive direction d. none of the above Overall Data: 19/40 = 48 % Students answered correctly 15/20 = 75 % of High-Performers answered correctly 4/20 = 20 % of Low-Performers answered correctly

8. Although the students’ overall performance on this question was low (only 48 % got it right), it accurately discriminated between the high-performers and the low-performers.

9. Example of Something Seriously Wrong #22. According to Kepler’s Second Law of Planetary Motion a planet in elliptical motion moves fastest when closest to the Sun and slowest when farthest from the Sun. The total Mechanical Energy of the planet-sun system is conserved. Which of the following must be true? I. Gravity does positive work for part of the orbit and negative work for part of the orbit. II. The total of Kinetic Energy and Potential Energy is constant. III. The Potential Energy of the system could be negative. a. I only b. II only c. III only d. II and III e. I, II, and III

10. Overall Data: 5/40 = 13 % Students answered correctly 3/20 = 15 % of High-Performers got it right. 2/20 = 10 % of Low-Performers got it right.

11. A Deeper Look Into this Data: Overall Data: Chose “A” 18 % (incorrect) Chose “B” 45 % (incorrect) Chose “C” 4 % (incorrect) Chose “D” 18 % (incorrect) Chose “E” 15 %

12. Looking at the most frequently wrong answers picked: Chose “B” 45 % This suggests that while students recognized that total energy was conserved (as was basically stated in the problem), they did not understand that: in an elliptical orbit gravity does both positive and negative work And 2. potential energy can be negative – choice of zero potential energy is arbitrary.

13. For This Particular Problem… We need to test the multiple concepts in this question with 2 or more separate questions. Questions can test more than one concept, but in this case it probably was not the best choice!

14. References… Popham, W. James (2003). Test better, teach better. Alexandria, Virginia: ASCD. Stiggins, R., Arter, J., Chappuis, J., Chappuis, S. (2006). Classroom assessment forstudent learning. Boston: Pearson.