Basic Testing Issues

1. 2007 Annual Conference Basic Testing Issues Ron Rodgers CTS/Employment Research Institute

2. Session Agenda A. Clarify test development Standards • Summarize test development process • Write & validate multiple choice items • Respond to your questions Council on Licensure, Enforcement and Regulation

3. APA Standards*(1954-1999) • APA Standards used by US Supreme Court to validate fair, valid examinations • Test must reflect job qualifications • Credentialing identifies qualified practitioners for public, employers • Panels of experts should help specify certification performance standards * Standards for Educational and Psychological Testing, American Psychological Association, American Educational Research Association, National Council on Measurement in Education Council on Licensure, Enforcement and Regulation

4. APA Standards*(1954-1999) • Candidates who fail receive report w/information about weak areas • Credentialing must depend on the knowledge and skills needed for acceptable professional performance • Criteria should not be adjusted to regulate the number or proportion of persons who pass the test Council on Licensure, Enforcement and Regulation

5. Define Exam Objectives • Create SME panel to represent experienced professionals in field • Identify references & initial specs • Conduct job analysis • Link test specifications to job tasks Council on Licensure, Enforcement and Regulation

6. Write & Validate Questions • Select item writing model • Draft item bank to meet test specs • Document each Q to references • Validate/edit Qs with SME panel Council on Licensure, Enforcement and Regulation

7. Score & Report Results • Assemble test forms • Set passing criteria • Review item performance statistics • Equate test forms • Adjust scoring if needed Council on Licensure, Enforcement and Regulation

8. Provide Candidate Info & Tech Support • Prepare candidate materials • Monitor test & item performance • Revalidate non-performing Qs • Amend items/test forms as needed Council on Licensure, Enforcement and Regulation

9. Maintain Test Security • Limit access to secure materials • Use non-disclosure agreements • Emphasize security to candidates • Define consequences of cheating Council on Licensure, Enforcement and Regulation

10. Non-Disclosure Agreement Each Subject Matter Expert (SME) acknowledges that all work performed under this Agreement for and on behalf of (program sponsor) is confidential and that no disclosure of any aspects of test data, information, materials and questions is permitted. The SME also agrees to adhere to all test security procedures and requirements specified by __________. Council on Licensure, Enforcement and Regulation

11. Responding to Your Questions • Writing good test questions Bruce Anderson, PES • Identifying your concerns Bruce Anderson, PES Ron Rodgers, CTS • Finding sources of assistance Council on Licensure, Enforcement and Regulation

12. Speaker Contact Information Ron Rodgers Continental Testing Service Employment Research Institute 809 Ridge Road, Suite 201 Wilmette, IL 60091 Phone: 847-256-5240 Email: rrodgers@Qmail.com Council on Licensure, Enforcement and Regulation

13. 2007 Annual Conference “Writing Good Multiple-Choice Items” Bruce Anderson Professional Examination Service

14. Agenda • Overview of the examination development process • Review of item-writing principles • Filling out the item-writing form • Review and development of items Council on Licensure, Enforcement and Regulation

15. Developing the Certification Examination • Conduct a job analysis/role delineation. • Generate test specifications. • Write items. • Validate/Improve items. • Enter items into item bank. • Assemble draft examinations. • Review and revise examination forms. Council on Licensure, Enforcement and Regulation

16. Developing the Certification Examination (contd.) • Conduct a passing point study. • Review item statistics. • Equate examination forms. • Score and report examination results. Council on Licensure, Enforcement and Regulation

17. What are the chances of an uninformed candidate selecting the correct answer to your item? 1 in 4, and that’s if you write a good item!! Council on Licensure, Enforcement and Regulation

18. Guessing the Correct Answer • For a multiple-choice item with four possible responses: • 3 good distracters = a 25% chance • 1 BAD distracter = a 33% chance • 2 BAD distracters = a 50% chance • 3 BAD distracters = a 100% chance of an uninformed candidate guessing the correct answer. Council on Licensure, Enforcement and Regulation

19. The Effects of Guessing on the Value of the Certification • The greater the ability of the uninformed candidate to guess correctly, the less valid and reliable the examination is. • The less valid and reliable the examination is, the less value is placed on the certification. Council on Licensure, Enforcement and Regulation

20. The Critical Objective when Writing Good Multiple-Choice Items • Reducing the ability of uninformed candidates to guess correctly by providing NO CLUES to the correct answer. Council on Licensure, Enforcement and Regulation

21. Where are the clues which help candidates to guess correctly? • The type of format chosen for the question • The format of the question asked (STEM) • The way the correct answer (KEY) is written • The way the incorrect answers (DISTRACTORS) are written Council on Licensure, Enforcement and Regulation

22. The Format of the Stem DIRECT QUESTION: Which of the following scientists was responsible for the development of the first polio vaccine? or INCOMPLETE STATEMENT: The first polio vaccine was developed by: • Jonas Salk • Louis Pasteur • Edward Jenner • Robert Koch Council on Licensure, Enforcement and Regulation

23. What is the capital of Kentucky? Frankfort* Lexington Louisville Ashland The major goal of raising the legal drinking age is to: Reduce driving fatalities and injuries.* Reduce liquor consumption. Discourage the habit of drinking by adolescents. Reduce roadside litter. Correct vs. Best Answer Stems Council on Licensure, Enforcement and Regulation

24. Guidelines for Constructing the Stem • Clarity and Directness: Is the requirement of the item clear without reference to the choices? • Sufficiency: Is enough information provided to answer the question? • Focus: Is the item focused on a single problem which assesses a meaningful fact or concept? Council on Licensure, Enforcement and Regulation

25. EXAMPLE: Clarity? Sufficiency? Focus? • The 2000 U.S. Presidential Election was decided by: • 1. fewer than 1000 votes • 2. the Electoral College* • 3. the Supreme Court • 4. December 21, 2000 Council on Licensure, Enforcement and Regulation

26. Guidelines for Constructing the Stem (cont’d) • Avoid negative stems. • Provide just enough information. • Do not use a word in the stem which only appears in the choice which is the key. • Do not provide gratuitous instruction. • Provide most information in the stem so that the options do not have repetition of words. Council on Licensure, Enforcement and Regulation

27. EXAMPLE: Negatives are confusing. • Which of the following authors is not a major American novelist? • 1. George Orwell • 2. William Faulkner • 3. John Steinbeck • 4. Ernest Hemmingway Council on Licensure, Enforcement and Regulation

28. EXAMPLE: Improved • Which of the following authors is a major British novelist? • 1. George Orwell • 2. William Faulkner • 3. John Steinbeck • 4. Ernest Hemmingway Council on Licensure, Enforcement and Regulation

29. Poor In 1850, Adolphe Chatin, the professor of pharmacy in Paris, believed that goiter resulted from an inadequate amount of iodine in the diet. The thyroid is part of which body system? Endocrine* Nervous Musculoskeletal Cardiovascular Better The thyroid is part of which body system? 1. Endocrine* 2. Nervous 3. Musculoskeletal 4. Cardiovascular EXAMPLE: Gratuitous instruction Council on Licensure, Enforcement and Regulation

30. Guidelines for the Correct Answer (KEY) • Is the intended key unambiguously correct and recognized as such by the experts? • Is the key parallel to the other choices (e.g., not shorter or longer, or more technical, or more detailed)? • Is there ONLY ONE clearly correct or best answer? Council on Licensure, Enforcement and Regulation

31. Guidelines for Distracters SOURCES OF DISTRACTORS • Common misconceptions • Common job performance errors • Statements which are true but which do not satisfy the requirements of the question • Statements which are either too broad or too narrow for the requirements of the question • Carefully worded incorrect statements which may sound plausible to the uninformed Council on Licensure, Enforcement and Regulation

32. Guidelines for The Choices • Does each choice follow logically and grammatically from the stem? • Are the choices parallel in: • length? • grammatical structure? • terminology? • content? • level of detail? Council on Licensure, Enforcement and Regulation

33. An American city noted for its production of automobiles is: 1. Detroit* 2. Ohio 3. Indiana 4. Philadelphia An American city noted for its production of automobiles is: Detroit* Cincinnati Indianapolis Philadelphia EXAMPLES: Parallelism Council on Licensure, Enforcement and Regulation

34. Serious communicable diseases that can cause developmental disability to the offspring of a pregnant mother are: 1. rubella and chicken pox* 2. cancer 3. rheumatism and arthritis 4. sinus infection A serious communicable disease that can cause developmental disability to the offspring of a pregnant mother is: rubella* cancer rheumatism sinus infection EXAMPLES: Parallelism, length. Council on Licensure, Enforcement and Regulation

35. Guidelines for Choices (contd.) • Do not repeat the same word in each answer choice; include the word once in the stem. As much time as possible is to be spent figuring out the answer rather than reading. • Make the choices mutually exclusive. • Do not use “all of the above” or “none of the above” as choices. The latter, for example, does not test whether a candidate knows the correct answer. Council on Licensure, Enforcement and Regulation

36. The right to vote in the United States is granted to individuals of what age? 1. 16 2. 17 3. 18* 4. 19 The right to vote in the United States is granted to individuals beginning at what age? 1. 16 2. 17 3. 18* 4. 19 EXAMPLES: Overlap Council on Licensure, Enforcement and Regulation

37. General Considerations • Is it RELEVANT to professional practice? • Is current BEST practice in the field tested? • Does it test an important concept of piece of knowledge? • Is it appropriate for entry-level? • Is the answer clear to a prepared candidate (not a ‘trick question’)? Council on Licensure, Enforcement and Regulation

38. General Considerations • Does it avoid controversial issues, e.g., religious, political, philosophical? • Are the context, setting, and content equally appropriate and familiar to all segments of the candidate population? • Is the terminology equally appropriate and familiar to all segments of the candidate population (e.g., international candidates)? • Is it free of bias re: persons or groups? Council on Licensure, Enforcement and Regulation

39. EXAMPLE: Sensitive Issue • Of the following profile analysis groups, the one most likely to have alcohol-related problems is: • 1. residents of rural areas in the South • 2. widowed males over 60 years of age • 3. women who are married and are wine drinkers • 4. men with no religious affiliation* Council on Licensure, Enforcement and Regulation

40. Other Sensitivity Issues • Avoid humor in test questions. • Avoid any subject that appears to be advancing a political or business agenda. • Avoid topics that could cause undue stress to the candidates, e.g., death, disease, war, brutality. • Avoid any stereotypes. Council on Licensure, Enforcement and Regulation

41. Rules of English • Don’t use no double negatives. • Make each pronoun agree with their antecedent. • Join clauses good, like a conjunction should. • Watch them participles when dangling. • About them sentence fragments. • Verbs has to agree with their subject. • Just between you and I, case is important too. • Don’t write run-on sentences they are hard to read. • Don’t use commas, that are not necessary. • Do’nt misuse apostrophes; its’ wrong. Council on Licensure, Enforcement and Regulation

42. THE I GNU TEST Council on Licensure, Enforcement and Regulation

43. 1. The purpose of the cluss in furmpaling is to remove: • 1. cluss-prags • 2. tremalis • 3. cloughs • 4. plumots Council on Licensure, Enforcement and Regulation

44. 2. Trassig is true when: • 1. lusp trasses the vom • 2. the viskal flans, if the viskal is donwill or zortil • 3. the belgo frulls • 4. dissles lisk easily Council on Licensure, Enforcement and Regulation

45. 3. The sigla frequently overfesks the treslum because: • 1. all siglas are mellious • 2. siglas are always votial • 3. the treslum is usually tarious • 4. no trelsa are feskable Council on Licensure, Enforcement and Regulation

46. 4. The fribbled breg will minter best with an: • 1. derst • 2. morst • 3. sortar • 4. ignu Council on Licensure, Enforcement and Regulation

47. 5. The mintering function of the ignu is most effectively carried out in connection with: • a razma tol • the groshing stantol • the fribbled breg • a frally sush Council on Licensure, Enforcement and Regulation

48. Problem Items Council on Licensure, Enforcement and Regulation

49. Problem Items • Which of the following persons was President of the United States when the Civil Rights Act of 1964 was enacted? • A. Lester Pearson • B. Anthony Eden • C. Jefferson Davis • D. Lyndon Johnson Council on Licensure, Enforcement and Regulation

50. Problem Items • Who orchestrated the December 7, 1941 attack on Pearl Harbor? A. Isoroku Yamamoto B. Chester Nimitz C. Bernard Montgomery D. Seiji Ozawa Council on Licensure, Enforcement and Regulation