Some perspectives on MCQ assessment

1. Some perspectives onMCQ assessment Stuart Palmer Mary Dracup Source: http://www.dreamstime.com/item.php?imageid=7299038

2. Assessment generally “Examinations are formidable even to the best prepared, for the greatest fool may ask more than the wisest man can answer.” Charles Caleb Colton (1780 - 1832)

3. General thoughts MCQs can be a useful element of an overall assessment design Can’t be abstracted from the context • No marks / diagnostic / formative / interactive • Few marks / semi-formative / assignment • High stakes / summative / examination Design and evaluation approach should be appropriate for the context

4. No need to look too far from home Considine, J; Botti, M and Thomas, S. Design, format, validity and reliability of multiple choice questions for use in nursing research and education. Collegian, 2005 Jan; 12 (1): 19-24.

5. A proposed MCQ process From: Considine, J; Botti, M and Thomas, S. Design, format, validity and reliability of multiple choice questions for use in nursing research and education. Collegian, 2005 Jan; 12 (1): 19-24.

6. An alternative/pragmatic process Design Content/Face validity Use Construct validity Reliability Keep / Replace

7. Design – unit assessment plan

8. Design – MCQ test plan

9. Design – NCLEX-RN MCQ test plan Source: 2010 NCLEX-RN® Detailed Test Plan

10. Face validity A sub-set of content validity Use an ‘expert panel’ to : • assess item clarity • assess item readability • assess consistency of style • check for errors

11. Construct validity Key check – do ‘experts’ agree on the correct answer? Item difficulty – the proportion of examinees that get the correct answer Distracter evaluation Item discrimination – the ability of an item to distinguish between knowledgeable students and others

12. Correlation with final unit mark r5 = 0.584 r6 = 0.569 (p < 5.3x10-10)(p < 1.8x10-9)

13. Correlation with final unit mark

14. Correlation with final unit mark 

15. Reliability Really ‘internal consistency’ Typically, Cronbach’s alpha or KR-20 Provides a measure of the average correlation among items within a test scale If a test examines a range of ‘themes’, may need to divide the test data into ‘scales’

16. Review of items Where items have been developed based on a systematic test plan and good content validity, they should only be discarded on the basis of clearly poor construct validity and reliability

17. Negative marking / formula scoring “Be the reasons for test-takers’ reluctance to answer under formula scoring what they may be, this reluctance is a psychometric Pyrrhic victory. Neither test-takers’ risk attitudes nor their strategic reasoning are what multiple-choice tests purport to measure. From the perspective of educational measurement, individual differences in people’s risk attitudes are nothing but unwanted noise: they add another source of variance, which is likely to reduce the test’s reliability and validity. Furthermore, the testmakers’ persisting inability to distinguish between a guessed correct answer and a known correct answer is compounded by their inability to distinguish between an omission deriving from ignorance and one deriving from risk aversion.” (Bar-Hillel, Budescu & Attali, 2005, p. 7)

18. Negative marking – discontinued by: The Royal College of General Practitioners – from 1992 • discriminated against female candidates The Royal College of Obstetricians and Gynaecologists – prior to 2004 • benefited bold and test-wise candidates; inhibited reasoning Examination Committee of the European Society of Anesthesiology – from 2008 • does not reflect real practice where decisions must be made on incomplete knowledge; no effect on rank ordering of results

19. A usefulresource? Source: https://www.ncsbn.org/1287.htm

20. Development of the NCLEX-RN Test Plan “A total of 12,000 newly licensed RNs are asked about the frequency and importance of performing 155 nursing care activities. Nursing care activities are then analyzed in relation to the frequency of performance, impact on maintaining client safety and client care settings where the activities are performed. This analysis guides the development of a framework for entry-level nursing practice that incorporates specific client needs, as well as processes fundamental to the practice of nursing.”

21. NCLEX-RN test plan structure Safe and Effective Care Environment • Management of Care • Safety and Infection Control Health Promotion and Maintenance Psychosocial Integrity Physiological Integrity • Basic Care and Comfort • Pharmacological and Parenteral Therapies • Reduction of Risk Potential • Physiological Adaptation

22. Design – NCLEX-RN MCQ test plan

23. Detailedtest plan

24. Detailedtest plan

25. Sample items – management of care For each of the eight content areas a sample item is provided

26. Steps in item writing • Select an area of the test plan for the focus of the item • Select a subcategory from the chosen area of the test plan • Select an important concept within that subcategory • Use the concept selected and write the stem • Write a key to represent important information the entry-level nurse should know • Identify common errors, misconceptions or irrelevant information • Use the previous information and write the distracters • Complete the item using the stem, key and distracters

27. Steps in item writing - example Step 1. Select an area of the test plan for the focus of the item • Safety and Infection Control Step 2. Select a subcategory from the chosen area of the test plan • Standard Precautions/Transmission-Based Precautions/Surgical Asepsis Step 3. Select an important concept within that subcategory • Evaluate infection control precautions implemented by staff members

28. Steps in item writing - example Step 4. Use the concept selected and write the stem • The nurse and nursing assistant are caring for a client with vancomycin-resistant enterococci (VRE). Which of the following activities by the nursing assistant would require immediate follow-up? Step 5. Write a key to represent important information the entry-level nurse should know • Contact Isolation: Assisting the client to ambulate in the hallway

29. Steps in item writing - example Step 6. Identify common errors, misconceptions or irrelevant information • Lack of understanding of isolation precautions • Uncertainty related to specific diagnosis Step 7. Use the previous information and write the distracters • Leaving a blood pressure cuff in the client’s room to be used by the client only • Putting on a protective gown to assist the client to sit in a chair • Taking the gloves off before leaving the client’s room

30. Steps in item writing - example Step 8. Complete the item using the stem, key and distracters The nurse and nursing assistant are caring for a client with vancomycin-resistant enterococci (VRE). Which of the following actions performed by the nursing assistant would require immediate follow-up from the nurse? • Leaving a blood pressure cuff in the client’s room to be used by that client only • Putting on a protective gown to assist the client to sit in a chair • Taking the gloves off before leaving the client’s room • Assisting the client to ambulate in the hallway (key)

31. Case scenario exercise Reduction of Risk Potential The nurse is caring for client who had a procedure three hours ago. Write an item that includes assessment data the nurse would observe in this client and which data should the nurse respond to first.

32. Reduction of Risk Potential – model The nurse is caring for a client who had a cardiac catherization 3 hours ago. Which of the following findings would be essential for the nurse to follow-up? • blood pressure increase from 103/68 to 110/70 over the past one hour • blood urea nitrogen (BUN), 22 mg/dL (key) • pulse, 101 • decrease in respiratory rate from 18 to 16 over the past one hour

33. DSO assessment tools - resources • Online assessment strategies: • http://www.deakin.edu.au/itl/dso/strategies-teaching/assessment.php • Blackboard assessment tools guides: • http://www.deakin.edu.au/itl/dso/guides/guide-list-assessments.php • Respondus guides: • http://www.deakin.edu.au/itl/dso/guides/guide-list-respondus.php

34. Case scenario exercise Psychosocial Integrity A nurse on an inpatient psychiatric unit observes a client pacing the hallway, mumbling and occasionally yelling aloud “Stop it!” Write an item describing the action the nurse should take in this situation.

35. Psychosocial Integrity – model MCQ The nurse is caring for a client who is mumbling, pacing in the hallway, and occasionally yelling “Stop it!” Which of the following actions should the nurse take? • Remove other clients from the area. • Escort the client back to the client’s room. • Request that the client be quiet and not disrupt others. • Use distraction to re-focus the client to reality. (key)

36. Thank you for your time