Designing effective multiple choice questions

1. Designing effective multiple choice questions Dr Andrew Oliver Learning & Teaching Institute University of Hertfordshire

2. Objective Testing • What are they? • Student provides a response to a question which has a pre determined answer • Which means • Select solution from a list of choices (MCQ) or • Enter short text or numerical responses (matched to correct answer)

3. Objective Tests • Diagnostic • Measure understanding in-class & adjust teaching, e.g. Electronic Voting Systems (EVS) • Formative • Provide ongoing feedback in-course and motivation, e.g. StudyNet • Summative • Exam type forming part of final course grade, e.g. QuestionMark

4. Advantages • Immediate feedback to students (and staff) • Decreased marking time • More frequent assessment • Provide statistics (improve question design) • STUDENTS LIKE IT ! (formative that is) • Any more?

5. Challenges • Time investment for design of questions & tests • As above but with technology • Measure high order skills ? • Yes – but time required to design • Difficult to assess essays (and hence creativity) • Well maybe, may be not • CHEATING • Network stability • Any more?

6. Question Design • Part 1: basic tips • Question stem • Question distractors • Part 2: Bloom’s taxonomy

7. Design: the question stem • single, definitive statement • avoid unnecessary or irrelevant material • use clear language • use negatives sparingly • put as much of the question in the stem • Any more?

8. Bad question I • Morphemes • a. are made up of smaller units called phenomes • b. are NOT found in grammatical function words • c. create meaning or have a grammatical function • d. can be described as free or bound

9. Good question I • Morphemes are the smallest contrastive unit of • a. compound words • b. prefixes • c. grammar • d. intransitive verbs • Clear, shorter, homogenous choices. • Provide single clear statement.

10. Bad Question II • Paul Muldoon, an Irish postmodern poet who uses experimental and playful language, uses which poetic genre in “Why Brownlee Left”? • a. sonnet • b. elegy • c. narrative poem • d. dramatic monologue • e. haiku

11. Good Question II • Paul Muldoon uses which poetic genre in “Why Brownlee Left”? • a. sonnet • b. elegy • c. narrative poem • d. dramatic monologue • e. haiku • Contains material irrelevant to the question. • Avoid unnecessary and irrelevant material.

12. Bad Question III • As mortality rates approach the zenith, what is the most likely ramification for the citizenry of an industrial nation? • a. an increase in the labour force participation rate of older people • b. a dispersing effect of population concentration • c. an upward trend in the youth dependency ratio

13. Good Question III • A major increase in mortality rates in an industrial nation is likely to cause • a. an increase in the labour force participation rate of older people • b. a dispersing effect of population concentration • c. an upward trend in the youth dependency ratio • Too complex – tests reading comprehension ? • Use clear, straightforward language

14. Bad Question IV • Which of the following is not a symptom of osteoporosis? • a. decreased bone density • b. frequent bone fractures • c. raised body temperature • d. lower back pain

15. Good Question IV • Which of the following is a symptom of osteoporosis? • a. decreased bone density • b. raised body temperature • c. hair loss • d. painful joints • Use negatives sparingly. • If needed then capitalize, underscore, embolden or otherwise highlight.

16. Bad Question V • Maslow’s theory of growth motivation asserts that • a. humanistic needs are physiological • b. humanistic needs are emotional • c. humanistic needs are intellectual

17. Good Question V • Maslow’s theory of growth motivation asserts that humanistic needs are • a. physiological • b. emotional • c. intellectual • Put as much of the question in the stem as possible, DON’T duplicate in each of the options

18. Design: the distractors • ensure there is only one correct answer • use plausible distractors • avoid clues to the correct answer • base distractors on common student errors (phenomenographics) • Any more?

19. Bad Question VI • Which of the following texts is considered to represent the pinnacle of modernist achievement? • a. The Waste Land • b. Middlemarch • c. “Ode to a Nightingale” • d. Ulysses • e. Ethan Frome

20. Good Question VI • Which of the following texts represents one of the high points of modernist achievement? • a. The Waste Land • b. Middlemarch • c. “Ode to a Nightingale” • d. Ethan Frome • e. “My Last Duchess” • ensure that there is only one correct response. • both options ‘a’ and ‘d’ could be considered to be correct.

21. Bad Question VII • The Dichotic Listening Test determines which side of the brain is directly involved in • a. tone perception • b. selection attention • c. cognition • d. hearing sounds

22. Good Question VII • The Dichotic Listening Test determines which side of the brain is directly involved in • a. tone perception • b. selection attention • c. pitch constancy • d. hearing sounds • Use only plausible and attractive alternatives as distracters. • ‘c’ not plausable

23. Bad Question VIII • A fertile area in the desert in which the water table reaches the ground surface is called an • a. mirage • b. oasis • c. water hole • d. polder

24. Good Question VIII • A fertile area in the desert in which the water table reaches the ground surface is called a/an • a. mirage • b. oasis • c. water hole • d. polder • Avoid giving clues to the correct answer • uses the article “an” which identifies choice “b” as the correct response.

25. Bloom’s Taxonomy est. 1956 • hierarchy of educational objectives • divides cognitive objectives into subdivisions • range from the simplest behaviour to the complex • other systems or hierarchies exist

26. Cognitive subdivisions • Knowledge • Comprehension • Application • Analysis • Synthesis • Evaluation

27. Bloom’s Taxonomy: Knowledge • Recall of information • Bringing to mind the appropriate information • Learning objectives: know facts, know theories, know terms, know events, know basic concepts, know places, know methods • Question cues: list, define, label, describe, name

28. Aim - to recall a specific date Question: in which year did the American Civil War end? a. 1832 b. 1865* c. 1857 d. 1888

29. Bloom’s Taxonomy: Comprehension • Ability to grasp meaning of the material – to interpret, to predict, to estimate • Learning outcomes: understand facts, interpret graphs, translate verbal to math formulae, estimate consequences from data. • Question cues: interpret, discuss, predict, summarise, classify

30. Aim - to understand and interpret definitions, relationships & analogies Question: which one of the following describes what takes place in the so-called PREPARATION stage of the creative process, as applied to the solution of a particular problem? 1. The problem is identified and defined. 2. All available information about the problem is collected. 3. An attempt is made to see if the proposed solution to the problem is acceptable. 4. The person goes through some experience leading to a general idea of how the problem can be solved. 5. The person sets the problem aside, and gets involved with some other unrelated activity. KNOWLEDGE is recalled and understanding (COMPREHENSION) tested of the meaning of each term (preparation).

31. Second Example: comprehension Question: In the following, a related pair of words is followed by five more pairs of words. Choose the response pair that best expresses a relationship similar to that expressed in the original pair. QUENCH : THIRST a. staunch : wound b. douse : fever c. antidote : poison d. extinguish : fire*

32. Bloom’s Taxonomy: Application • Ability to use learned materials in NEW situations • Application of rules, methods, concepts and theories. • Learning outcomes: apply principles / theories to new situations, solve math problems, construct graphs. • Question cues: apply, demonstrate, show, relate, calculate

33. Aim: to calculate velocity Question: End A of the cord is moving 6m/s to the left. Compute the velocity of the block B and select one of the responses below. a. 2 m/s (to left) b. 3 m/s (to left) c. 6 m/s (to left) d. 12 m/s (to left) e. none of the above Nb there would be a diagram here!

34. Second Example: application Which one of the following values approximates best to the volume of a sphere with radius 5m? • 2000m³ • 1000m³ • 500m³ • 250m³ • 125m³

35. Bloom’s Taxonomy: Analysis • Ability to break down material into components and understand patterns (underlying structure) • Recognition of parts and their relationship to each other • Learning objectives: recognise unstated assumptions, evaluate the relevancy of data • Question cues: analyse, arrange, order, explain, connect, infer, compare, categorize

36. Aim: to analyse and infer from a geological map Question 1. LANDSLIPS At which of the following contacts between strata would you expect MOST landslips to occur? Choose 2 of the following options. a. Chalk above Gault Clay b. Gault Clay above Lower Greensand c. Corallian limestone above Oxford Clay d. Cornbrash limestone above inferior Oolite limestone

37. Bloom’s Taxonomy: Synthesis • Use old ideas to create new ones • Organise and relate knowledge from several areas • Learning objectives: writing themes & stories, formulating a new scheme for classifying objects (or events, or ideas). • Question words: integrate, modify, invent, design, compose, plan, formulate, arrange

38. Aim: to organise and arrange appropriate critical terms in order to construct a geological analysis of the following photographic image Question: Move the appropriate descriptive terms from the list to the ‘form’ and ‘attitude’ boxes below. Question created by Professor Don MacKenzie, TRIADS.

39. Bloom’s Taxonomy: Evaluation • Make judgements of / evaluate data • Assess value of ideas, theories • Compare and discriminate between ideas • Learning outcome: judge the adequacy with which conclusions are supported by data • Question cues: appraise, evaluate, defend, rank, conclude, discriminate, recommend

40. Aim: to assess the condition of a patient based on various readings Question: An adult subject breathing air was found to have the following lung volumes: • Vital capacity - 3.5 litres • Forced expiratory volume in 1 sec (FEV1) - 2.8 litres • Functional residual capacity (FRC) - 1.8 litres • Residual volume (RV) - 0.8 litre True / False a. There is no obstruction to airflow. b. The subject must be abnormal. c. The expiratory reserve volume is 1 litre. d. All of these measurements could have been made using only a spirometer. e. There would be approximately 250 ml of oxygen in this subject’s lungs at the end of a tidal expiration. Question by Dr David A. Davies, The University of Birmingham (MEDWEB Computer- Assisted Assessment web site - http://medweb.bham.ac.uk/caa)

41. References I • Designing and Using Objective Tests, Dr Joanna Bull, The CAA Centre, 2001 • Other resources : • Designing and Managing Multiple Choice Questions, Carneson, J., Delpierre, G. & Masters, K., University of Cape Town • Measurement and Evaluation in Teaching, Gronlund, N. E., 1971 (In the library)

42. References II • StudyNet online tutorials (animated) • Log in StudyNet • Technical Support section • Online Tutorials (under ‘Using StudyNet’) • Covers ‘how to’ and includes examples, statistics info

43. ‘How to’ series • How to Blog in 5 minutes (Word) • How to Wiki in 5 minutes (Word) • How to create a Podcast in 5 minutes (Word) • How to convert audio files to MP3 in 5 minutes (Word) • How to use RSS in 5 minutes (Word)

44. ‘You need to know…’ series • You need to know about Podcasting (Slides) • You need to know about Podcasting: a teaching and learning guide (Word) • You need to know about Online Discussions: a teaching and learning guide (Word) • You need to know about Blogs (Slides) • You need to know about Wikis (Slides) • You need to know about RSS (Slides) • You need to know about Web 2.0 (Slides) • You need to know about Computer Assessment (Slides) • You need to know about Effective Question Design (Slides) • You need to know about Electronic Voting Systems (Slides)

45. Also available: • Podcasting: audio talk on the benefits of podcasting (hi quality) • Computer Assessment: using statistic analysis to design effective questions For copies please contact: Dr Andrew Oliver. LTDU, College Lane LRC (a.oliver@herts.ac.uk, ext. 4754).