Writing Multiple Choice Tests Presentation to First Year Biology Educators’ Colloquim

Writing Multiple Choice TestsPresentation to First Year Biology Educators’ Colloquim Professor Jeffrey Smith University of Otago

The Basic Approach • Step One: The test blueprint – what do you want to test? What is the content and what is the level of thinking you want to assess? • Is the test blueprint tightly related to the course outline/objectives? • Do the students know what you expect of them?

Levels of sophistication in thinking(Bloom’s taxonomy) • Knowledge • Comprehension • Application • Analysis • Synthesis • Evaluation

The Basic Approach • Step Two: Logistics. • How much time is available for the test? • What kinds of items will best fit your purpose? • Consider marking in addition to optimal format from a measurement perspective. (Might a multiple choice format save you a lot of time down the road that could be used in more productive activities than marking essays?) • How many items and of what type? (Rule of thumb is a minute per item for MC, but this is easy to try out.)

The Basic Approach • Step 3: Align your items with your test blueprint. • What kinds of items are you going to use where? • Are you covering all of the content that is important or just the content where it is easy to write items? (Beware of the tyranny of the readily quantifiable.)

The Basic Approach • Step 4: (Isn’t this fun?) Now you have your test outlined and items assigned to the various components of the test. So what is next? • Draft the items.

Writing the Multiple Choice Item • What do you want to know about the student with this item? What would a great item do here? • My approach tends to focus a lot on knowledge, comprehension, and application from Bloom’s taxonomy. Especially when writing MC items.

Writing the Multiple Choice Item • Write the stem first. • End it in a question mark. • Pose a clear problem. • Don’t worry about the choices right now. Read the item. Think about what you are wanting to measure from your test blueprint. How good a match do you have?

Sample knowledge level item(What does it require?) • What is the space between the end of the axon of one neuron and the beginning of the dendrite of another neuron called? • A. cortex • B. corpus callosum • C. glial cell • D. synapse

Sample knowledge level item 2(What does it require?) • Which lobe is associated with higher order cognitive functioning, such as judgement and problem-solving? • A. frontal lobe • B. parietal lobe • C. temporal lobe • D. occipital lobe

Sample comprehension level item (What does it require?) • Which of the following best describes the transmission of information from one neuron to another? • A. The axon of one neuron is attached to the dendrites of a number of postsynaptic neurons, which lets electrical current be passed amongst neurons. • B. An electrical impulse down the axon of one neuron causes neurotransmitters to be sent across a synaptic gap and received by dendrites of another neuron. • C. The nucleus of the first cell divides, and the information contained in the DNA of that cell passes on to the DNA of the second cell through the synaptic cleft. • D. The axon of one neuron sheds its outermost glial cell, which is received by the postsynaptic neuron through the transmitters contained at the end of the dendrites of the receiving cell.

Sample comprehension level item 2(What does it require?) • Which of the following best describes a social constructivist perspective? • A. The process of learning moves from the external to the internal. • B. The process of learning is facilitated by trying to make sense of the world. • C. The process of learning involves taking in information and effectively processing it. • D. The process of learning is not linear, but occurs sporadically over time.

Sample application level item (What does it require?) • Simon has moved from the United States to New Zealand. He plays softball well but has never played cricket. When invited to take a turn at bat, he tries to adapt the way he would swing with a softball bat to swinging a cricket bat. What would Piaget call this behaviour? • A. Elaboration • B. Assimilation • C. Rehearsal • D. Accommodation

Sample application level item 2(What does it require?) • Mrs White is working with her Year 4 children on the properties of magnets. She asks what a magnet does and Molly answers, “It can pull things.” Mrs White responds, “Let’s test that idea.” She gets out a cotton ball, a pencil, a piece of string, and a bit of clay and tries to pull them with the magnet. When none of them work, she asks what the problem is. Molly says, “That magnet is broken.” Steve says, “Maybe it isn’t a magnet.” What is Mrs White doing with the class? • A. Trying to move the students into a formal operational way of thinking. • B. Trying to set up disequilibrium in the environment of the students. • C. Trying to encourage the students to recall information stored in long term memory. • D. Trying to get the students to adopt a hypothesis-testing approach to science.

Drafting items • Knowledge level • What is x called… • Which of the following is associated with… • Who did x… • How many ….

Drafting items • Comprehension level • Which is the best description of… • (Or, what does this describe?...) • What is the main advantage of… • What kind of …. • (Or, according to theory x, which…) • The major difference between x and y is….

Drafting items • Application level • Starting with the setting or vignette: • Daniel is teaching his son Blake how to ride a bicycle. In the beginning, he ran alongside Blake and helped him balance the bike. Then, he let go for a few moments to let Blake do the balancing himself. Now, Daniel is only holding on at the very beginning. What is Daniel engaged in? • Mr Chapman wants his chemistry class to learn the periodic table of elements. He wants the students to understand the basics of the molecular makeup of the elements and how they are related to one another. He realises that this is a large task for the students, so he is trying to come up with ways to help them in their learning. Which model of learning would be the most helpful here?

Right answers • Your students will be nervous when they take the test. Make sure your right answers are right and your wrong answers are wrong. Nobody likes “gotcha”! • Seriously, nobody does. • Write the right answer first. Make sure it is a clear and direct response to the question as posed, unambiguous, and correct.

Wrong answers • Wrong answers determine how difficult an item is. • Toby is using a new computer program to help him learn to read. The program is set up so that whenever Toby sees a word he cannot pronounce, he can scroll over it with the mouse. The word is then pronounced out loud by the computer. What is this an example of? • A. Mediation A. Meditation • B. Transfer B. Trans Alpine • C. Web-based inquiry C. Moose doots • D. Scaffolding D. Scaffolding

Wrong answers • Wrong answers need to be: • Clearly wrong • Reasonable given the context • About the same length as right answers • Written as carefully and cautiously as right answers • Letter C sometimes • As far away from being right as you want the item to be difficult

The Basic Approach • Step 5 (Remember the steps?): Review, edit, revise • You are a scholar in the general area of biology, not test construction • I, on the other hand, am a scholar in the area of test construction • I still screw this up all the time • Thus, we review, edit, and revise. Get a colleague to read your test and give you feedback. (It won’t hurt too much.) • Get your partner to do so (mine happens to be a scholar in the area of testing, so that’s kind of a bonus – sometimes) • Remember that your students will be nervous and will try to overthink this stuff.

The Basic Approach • Step 6: • Go back to your test blueprint and make sure your test measures what you want it to.

The Basic Approach • Step 7: • Take your test as if you were a student. This helps for: • Timing • Determining difficulty • Final adjustments

That’s It! • (If I could have come up with five more steps, you could become recovering psychometricians!) • Thanks!

Writing Multiple Choice Tests Presentation to First Year Biology Educators’ Colloquim