1 / 23

Establishing Passing Standards Without Gambling

Establishing Passing Standards Without Gambling. Photo by Mel Curtis. Monitor on Psychology, September 2002. What is a Standard?. Cut Score = Test score a candidate must attain to pass Raw Score - for example, 36 correct Percent Correct - 36/50 = 72% Scaled Score - for example, 300.

thomasjwilliams
Télécharger la présentation

Establishing Passing Standards Without Gambling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Establishing Passing Standards Without Gambling Photo by Mel Curtis. Monitor on Psychology, September 2002.

  2. What is a Standard? Cut Score = Test score a candidate must attain to pass Raw Score - for example, 36 correct Percent Correct - 36/50 = 72% Scaled Score - for example, 300 Standard = The level of knowledge or proficiency a candidate must demonstrate to pass

  3. Why Do We Need a Formal Procedure? 70%? • Defensibility • Fairness • Validity

  4. Important Factors in Setting a Standard • Subject Matter Experts • A Representative Committee • Ample Time • Actual Test Items (Questions) • Definition of the Level of Proficiency Distinguishing Qualified and Unqualified Candidates

  5. Some Common Methodsof Standard Setting • Angoff • Item Mapping • Bookmark • Contrasting Groups

  6. The (Modified) Angoff Method Subject Matter Experts define the minimally competent (borderline) candidate in terms of knowledge. They evaluate every item in the test and estimatethis candidate’s chances of answering correctly. The mean estimate across all experts and all items determines the passing score.

  7. Steps in a Typical Angoff Session Take the Test! Define the "Just Sufficiently Qualified" (JSQ) or Minimally Competent (MC) Candidate What is the Probability the JSQ Candidate Will Answer an Item Correctly?

  8. Steps in Angoff, continued Determine Probabilities in Groups of 10 Items - Discuss Change Probability Estimates if Desired Add Mean Ratings for All Items to Calculate Cut Score Evaluate: Is This Cut Reasonable?

  9. Advantages of the Angoff Method • A relatively straightfoward process • No data necessary • Has held up in court

  10. Disadvantages of the Angoff Method • Must look at every item on test(s) • Time and cost • Fatigue, inattention, “rushing” • Difficulty of accurately estimating probabilities

  11. Item Mapping A graphical method of determining the level of competence necessary for licensure

  12. Item Mapping Process Administer items to a pilot group.Collect statistics, including the difficulty of each item. Group items by difficulty. Display in a graph.Subject Matter Experts define the minimally competent (borderline) candidate.SMEs evaluate a sample of items: Does the borderline candidate have at least a 50% chance of answering correctly? Evaluate: Is this cut reasonable?

  13. Rasch Model if Candidate Ability = Item Difficulty then Chance of a correct answer = 50%

  14. Item Mapping Advantages • Sound statistical basis • More discussion (no “rushing”) • Portrait of the borderline candidate • Multiple forms cut simultaneously • Time Disadvantages • Less straightforward (Rasch model) • Requires empirical data

  15. Bookmark MethodConceptually Similar to Item Mapping Administer items to a pilot group.Collect statistics, including the difficulty of each item. Order items by difficulty. Display in a booklet.Subject Matter Experts define the minimally competent (borderline) candidate.

  16. Bookmark Method, continued SMEs review items and place a bookmark between items the minimally acceptable candidate is likely to answer correctly and items this candidate is unlikely to answer correctly. Discuss and repeat the process, aiming for agreement. Evaluate: Is this cut reasonable?

  17. Bookmark Method Advantages • More discussion (no “rushing”) • Portrait of the borderline candidate • More focus on item content over entire exam • High level of face validity Disadvantages • Tends to be time-consuming • Requires empirical data

  18. Contrasting Groups Administer items to a pilot group. Subject Matter Experts classify each candidate as qualified or unqualified based on other data. Score the exam and order candidate IDs by score. Find a score or a narrow range of scores for whichapproximately half of the candidates have been labeled unqualified.

  19. Contrasting Groups Score Number of Candidates Percent Qualified / Unqualified Qualified 46-50 5 0 100 41-45 14 1 93 36-40 25 7 78 31-35 22 10 69 26-30 17 12 59 21-25 11 12 49 16-20 4 12 33 11-15 0 6 0 0-10 0 1 0

  20. Contrasting Groups • Not widely used in licensure testing • Subjectivity of judgments (Q or UnQ) • Connection to job is less direct • Often not feasible to get judgments

  21. ?

More Related