Writing Exam Questions in the Clinical Sciences

1. Writing Exam Questions in the Clinical Sciences Faculty Professional Development SeriesUniversity of Pennsylvania School of MedicineNovember 17, 2004 Jennifer R. Kogan, M.D. Judy A. Shea, Ph.D. Department of Medicine Materials adapted from the National Board of Medical Examiners

2. Agenda Review item types Review structure of A type questions Review technical item flaws Analyze submitted items Wrap-up

3. Steps in Test Development Test purpose Testing time and method of administration Test standardization Test content Item format Number of items Developing items Item selection and evaluation

4. Overview of Item Types True-false • C (A/B/Both/Neither) • K (complex true/false) • X (simple true/false) • simulations such as PMPs One-best answer • A (4 or more options) • B (4 or 5 option matching sets in sets of 2–5 items) • R (extended matching items in sets of 2-20 items)

5. True/false question X-type Which of the following is/are X-linked recessive conditions? 1. Hemophilia A (classic hemophilia) T 2. Cystic fibrosis F 3. Duchenne’s muscular dystrophy T 4. Tay-Sachs disease F

6. Advantages simple direct test of knowledge efficient easy to write Disadvantages each statement must be clearly true/false test trivia encourage memorization ambiguous susceptible to guessing True-false Questions Avoid true-false questions

7. Single Best Answer A-type A previously healthy 15-year-old boy has cramping periumbilical pain; after several hours, the pain shifts to the right lower quadrant and becomes constant. He vomits several times and is brought to the emergency department. The abdomen is tender on deep palpation of the right lower quadrant. Findings on chest and abdominal x-ray films are normal. Leukocyte count is 15,000/mm3. Urinalysis shows 3 leukocytes/hpf. Which of the following is the most appropriate initial management? A. Supportive treatment at home; return at once if the pain increases B. Barium enema C. CT scan of the abdomen D. Intravenous pyelography and cystography E. Surgical exploration of the abdomen*

8. Extended Matching Question R-type A. Left anterior cerebral artery E. Left posterior cerebral artery B. Right anterior cerebral artery F. Right posterior cerebral artery C. Left middle cerebral artery G. Left lenticulostriate arteries D. Right middle cerebral artery H. Right lenticulostriate arteries For each patient with neurological abnormalities, select the artery that is most likely to be involved. 1. A 72-year-old right-handed man has weakness and hyperreflexia of the right lower limb, an extensor plantar response on the right, normal strength of the right arm, and normal facial movements. Answer: A 2. A 68-year-old right-handed man has right spastic hemiparesis, an extensor plantar response on the right, and paralysis of the lower two-thirds of his face on the right. His speech is fluent, and he has normal comprehension of verbal and written commands. Answer: G

9. Item Type Answer Implications True-false • absolute/non-debatable One-best answer • there is one better/best answer True False C A D E B

10. Components of the A-type Question Stem:A 65-year-old man has difficulty rising from a seated position and straightening his trunk, but he has no difficulty flexing his leg. Lead-in:Which of the following muscles is most likely to have been injured? Options:A. Gluteus maximus* B. Gluteus minimus C. Hamstrings D. Iliopsoas E. Obturator internus Distractors

11. Rules for Writing A-type Questions 1) Focus on an important topic, usually a common or critical clinical problem; avoid esoterica and “zebras” 2) Assess application of knowledge, not recall 3) Pose clinical decision-making tasks that are within the education/experience of examinees 4) Pose a clear question in the lead-in • can you answer it without looking at the options? 5) Use homogeneous distractors 6) Avoid technical flaws

12. Patient vignettes should include age, gender site of care presenting complaint duration patient history physical findings +/- diagnostic studies +/- initial treatment Stems should not be completely based on real patients include reference material when it would be realistic in practice not use the patient’s or doctor’s own words not include patients who lie Tools

13. Lead-Ins Health maintenance • Which of the following is the most appropriate screening test? • Which of the following immunizations should be administered at this time? Mechanisms of disease • Which of the following is the most likely pathogen? • Which of the following is the most likely explanation for the findings?

14. Diagnosis • Which of the following is the most likely diagnosis? • Which of the following is the most appropriate next step in diagnosis? Management • Which of the following is the most appropriate next step in patient care? • Which of the following is the most effective management?

15. Technical Item Flaws Issues related to Irrelevant Difficulty Issues Related to Testwiseness

16. Grammatical Cues The option(s) does not flow from the stem The minor differences among organisms of the same kind are known as A. Heredity B. Variations C. Adaptation D. Natural selection

17. Logical Cues A subset of options are collectively exhaustive. Crime is A. Equally distributed among the social classes B. Overrepresented among the poor C. Overrepresented among the middle class and rich D. Primarily an indication of psychosexual maladjustment E. Reaching a plateau of tolerability for the nation

18. Absolute Terms Terms such as ‘always’ or ‘never’ are used in options. In patients with advanced dementia, Alzheimer’s type, the memory defect A. Can be treated adequately with lecithin B. Could be a sequela of early parkinsonism C. Is never seen in patients with neurofibrillary tangles C. Is never severe D. Possibly involves the cholinergic system

19. Long Correct Answer The correct answer is longer, more specific, or more complete than the other options. Secondary gain is A. Synonymous with malingering B. A frequent problem in obsessive-compulsive disorder C. A complication of a variety of illnesses and tends to prolong many of them D. Never seen in organic brain damage

20. Word Repeats A word or phrase is included in the stem and correct answer. A58-year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called A. Depersonalization B. Derailment C. Derealization* D. Focal memory defect

21. Convergence The correct answer includes the most elements in common with the other options Local anesthetics are most effective in the A. Anionic form, acting from inside the nerve membrane B. Cationic form, acting from inside the nerve membrane* C. Cationic form, acting from outside the nerve membrane D. Uncharged form, acting from inside the nerve membrane E. Uncharged form, acting from outside the nerve membrane

22. Options are long, complicated or doubled Systematic geography differs from regional geography in that A. Systematic geography deals, in the main, with physical geography, while regional geography concerns itself essentially with the field of human geography B. Systematic geography studies a region systematically while regional geography is concerned only with descriptive account of a region C. Systematic geography studies a single phenomenon in its distribution over the earth in order to supply generalizations for regional geography, which studies the arrangement of phenomena in one given area*

23. Numeric data are not stated consistently Following a second episode of infection, what is the the likelihood that a woman is infertile? A. Less than 20% B. 20% to 30% C. Greater than 50% D. 90% E. 75%

24. Frequency terms in the options are vague Severe obesity in early adolescence • Usually responds dramatically to dietary regimens • Often is related to endocrine disorders • Has a 75% change of clearing spontaneously • Shows a poor prognosis • Usually responds to pharmacotherapy and intensive psychotherapy

25. Language in the options is not parallel In a vaccine trial, 200 2-year-old boys were given a vaccine against a certain disease and then monitored for five years for occurrence of disease. Of this group, 85% never contracted the disease. Which of the following statements concerning these results is correct? • No conclusions can be drawn since no follow-up was made of non-vaccinated children • The number of cases (I.e. 30 cases over five years) is too small for statistically meaningful conclusions • No conclusions can be drawn because the trial involved only boys • Vaccine efficacy (%) is calculated as 85-15/100

26. Fixed In a vaccine trial, 200 2-year-old boys were given a vaccine against a certain disease and then monitored for five years for occurrence of disease. Of this group, 85% never contracted the disease. For which of the following reasons can no conclusion be drawn from these results? • No follow-up was made of non-vaccinated children • The number of cases (I.e. 30 cases over five years) was too small • The trial involved only boys • Write new option

27. Options in an nonlogical order The population of Denmark is A. 2 million B. 15 million C. 4 million D. 7 million

28. “None of the Above” is used as an option Which city is closest to New York City? A. Boston B. Chicago C. Dallas D. Los Angeles E. None of the above

29. “Window Dressing” and “Red Herrings”

30. The most likely renal abnormality in children with nephrotic syndrome and normal renal function is Acute poststreptococcal glomerulonephritis Hemolytic-uremic syndrome Minimal change disease Focal and segmental glomerulosclerosis Schonlein-Henoch purpura Non-vignette

31. A 2 year old boy has a 1 week history of edema. His blood pressure is 100/60 mmHg and there is generalized edema and ascites. Labs show Cr 0.4 mg/dL, albumin 1.4 g/dL and cholesterol of 569 mg/dL. UA shows 4+ protein and no blood. The most likely diagnosis is Short vignette

32. A 2 year old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mmHg, pulse 110/min respirations 28/min. Exam shows swelling of eyes, abdominal distention and a positive fluid Labs show Cr 0.4 mg/dL, albumin 1.4 g/dL and cholesterol of 569 mg/dL. UA shows 4+ protein and no blood. The most likely diagnosis is Long Vignette

33. Analysis of Submitted Items

34. Start with the theme Fatigue List all diagnoses Acute leukemia Congestive heart failure Depression Epstein-Barr virus Folate deficiency Glucose 6-phosphate deficiency And so on… Write vignettes A 15-year-old girl has a two week history of fatigue and back pain. She has widespread bruising, pallor, and tenderness over the vertebrae and both femurs. Complete blood count shows hemoglobin concentration of 7.0 g/dL, leukocyte count of 2000/mm, and platelet count of 15,000/mm Steps in Writing R-types

35. Some Tips for Writing R-types More than one vignette can be prepared for common treatable conditions Shorter/longer vignettes can be used It is easy to convert them to one-best-answer questions and vice versa There needs to be a single best answer for each stem All rules/technical flaws for single-best answer questions apply

36. Sample Lead-ins and Topics for R-types For each of the following patients, select the most likely (cause). • Underlying mechanism of disease, medications, toxic agents… For each of the following patients with (chief complaint), select the most likely diagnosis. • Lists of diagnoses For each of the following patients, select the (finding) that would be expected. • Laboratory results, physical signs…

37. Take Home Points • Each item should focus on an important concept • Each item should assess application of knowledge, not recall of an isolated fast • The stem of the item must pose a clear question, and it should be possible to arrive at the answer with the options covered • All distractors should be homogeneous • Avoid technical item flaws that provide specific benefit to testwise examinees or that pose irrelevant difficulty.

38. Suggested Reading • Basic texts • Case, .S and Swanson, D.B. Constructing Written Test Questions for the Basic and Clinical Sciences, NBME, 2003. http://www.nbme.org/PDF/ItemWriting_2003/2003IWGwhole.pdf • Crocker, L and Algina, J. Introduction to Classical and Modern Test Theory. Holt, Rinehart and Winston, 1986. • Ebel, R.L. Measuring Educational Achievement. Prentice Hal, 1965. • Haladyna, T.M. Developing and Validating Multiple-Choice Test Items. Laurence Erlbaum Associates, Inc. • Advanced texts • Baker, F.B. The Basics of Item Response Theory. Heinemann, 1985. • Linn, R.L. (Ed.) Educational Measurement. Macmillan Publishing Company, 1989. • Wainer, H. and Braun, H.I. (Eds.) Test Validity. Lawrence Erlbaum, 1988.