1 / 51

Examination Accreditation Program

Examination Accreditation Program. Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA. Put to the Test: Introducing the Presenters. Critical Issues Lawrence J. Fabrey, Ph.D. Content Development Tadas Dabsys. Writing Exam Questions

maxim
Télécharger la présentation

Examination Accreditation Program

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA

  2. Put to the Test: Introducing the Presenters Critical Issues Lawrence J. Fabrey, Ph.D. Content Development Tadas Dabsys Writing Exam Questions James F. Fryer, Ed.D., CPCU Performance Analysis Nikki Eatchel M.A.

  3. Critical Issues in Examination Development • Validity • Reliability • Fairness • Legal Defensibility • Standards

  4. Critical Issues: Validity • Chapter 1 in the Standards • Standards for Educational and Psychological Testing (AERA/APA/NCME, 1999) • 24 separate standards • Also in Chapter 14 – Testing in Employment and Credentialing • Especially 14.8 through 14.10: Explicit definition, link between job and test content, rationale basis

  5. Critical Issues: Validity • We cannot claim an examination is “valid” • “Links in the chain of evidence used to support the validity of the examination results” • Links include: • Job Analysis, Test Specifications, Item Writing, Standard Setting, and Examination Construction, Administration and Scoring.

  6. Critical Issues: Validity Traditional Sources of Validity Evidence • Content • Criterion Related (Predictive or Concurrent) • Construct Now -- Validity is a unitary concept: The degree of accumulated evidence is key

  7. Critical Issues: Validity Current (1999 Standards) – Validity Evidence based on: • Test Content • Response Processes • Internal Structure • Relation to Other Variables • Consequences of Testing

  8. Critical Issues: Validity For Licensing, Validity Evidence Based on Test Content is Critical • Job analysis provides the foundation • Identifies job-related activities/tasks (and sometimes KSAs) that will be used to define test content

  9. Critical Issues: Reliability • Chapter 2 in the Standards • 20 distinct standards (some not relevant for RE licensing exams) • Reliability refers to the consistency of measurement (reproducibility)

  10. Critical Issues: Reliability • All scores (all measurements) have a component of error • Scales, equivalence

  11. Critical Issues: Reliability • In general, four factors increase reliability: 1. Measurement of homogeneous content 2. Heterogeneous candidate groups 3. Longer tests 4. High quality items

  12. Critical Issues: Reliability • Estimates of reliability to be derived depend on measurement model: • Classical Test Theory (CTT) • Item Response Theory (IRT)

  13. Critical Issues: Reliability • Classical Test Theory • Unit of measurement: based on # correct • Attempt to estimate “true score” • Estimates of reliability focus on consistency of scores

  14. Critical Issues: Reliability • Item Response Theory • Based on probabilities of correct responses • Attempt to estimate probability based on underlying trait (or ability) parameter • Estimates of reliability focus on information obtained

  15. Chapter 7 in the Standards Four principle aspects of fairness: Lack of bias Equitable treatment in testing process Equality in outcomes Opportunity to learn Critical Issues: Fairness

  16. Lack of bias Technical definition: Bias is present if different meanings of scores are present for identifiable subgroups Differential item functioning (DIF) Judgmental and statistical processes Content-related sources Response-related sources Critical Issues: Fairness

  17. Equitable treatment in testing process Equal opportunity to demonstrate knowledge Standardized testing conditions Security (before, during, after) Equal opportunity to prepare Candidate materials Critical Issues: Fairness

  18. Equality in outcomes Are equal passing rates for identifiable subgroups required? No, what is required is that all outcomes must have the same meaning Equality of the passing point Critical Issues: Fairness

  19. Opportunity to learn Primarily applicable to educational achievement testing Critical Issues: Fairness

  20. Highlights of Chapters 8, 9, and 10 8: Rights and Responsibilities of Test Takers – focus on candidates’ rights (e.g., to information) 9: Testing Individuals of Diverse Linguistic Backgrounds – issues of translation, adaptation, modification and potential impact on validity 10: Testing Individuals with Disabilities – ADA requires “reasonable accommodations” Critical Issues: Fairness

  21. Critical Issues: Legal Defensibility • All Standards could be considered • Standards are not legal requirements • New Standards under development

  22. Critical Issues: Legal Defensibility Other considerations • ARELLO Guidelines for Accreditation • EEOC Guidelines (1978)

  23. Put to the Test: Content Development Tadas Dabsys

  24. Exam Content Development • Job Analysis Survey • Sampling Methodology • Analysis of the Survey Results • Content Specifications

  25. Job Analysis Survey • Job Analysis • Procedure • Content Development • Format Development

  26. Sampling Methodology • Sampling – Representative Sample • Target Groups • Sources for Addresses/Contact Information

  27. Analysis of the Survey Results • Demographic Information • Professional Activities • Knowledge • Rating and Respective Response Scales • Summary Statistics • Identification of Qualifying Professional Activities/Knowledge

  28. Content Specifications • Development of Content Specifications • Establishing Content Base for Inclusion in Exams • Linking Knowledge Areas to Professional Activities • Development of Operational Definitions • Test Outline and Content Weighting

  29. Put to the Test: Writing Exam Questions James F. Fryer, Ed.D., CPCU

  30. Developing the Exam Question(Item Writing) How? • National Subject Matter Experts • Qualifications • Experience • Background in Job Analysis

  31. Developing the Exam Question(Item Writing) Linking Back to the Job Analysis: • Activities and Tasks • Subject/Content Area of the Task • The Knowledge Statement (KSAs)

  32. Developing the Exam Question(Item Writing) Linking to the Knowledge Statement: Content Topic Area – Leasehold Estates Content Sub-Topic Area – Basic Elements of a Lease Job Task – Negotiate a Lease KSA – Knowledge of the financial ramifications of the common lease provisions referred to as Net Lease, Gross Lease, Triple-Net Lease, Percentage Lease, Base Rent, Effective Rent

  33. Developing the Exam Question(Item Writing) Linking to the Knowledge Statement: Knowledge Statement – The entry level practitioner must be able to compare and contrast the interrelationships of the commonly negotiated lease provisions referred to as Net Lease, Gross Lease, Triple-Net Lease, Percentage Lease, Base Rent, Effective Rent and equate the comparison in terms of financial impact on cost to the landlord or tenant.

  34. Developing the Exam QuestionBalancing Cognitive Levels: Recall: Able to state the definition of a “latent defect.” Application: Applies knowledge of the definition of “latent defect” by recognizing and classifying certain described property conditions as meeting the definition of a latent defect. Analysis: Applies knowledge of the definition by recognizing certain described property conditions as meeting the definition and interpreting the information to determine what action should be made.

  35. Developing the Exam Question(Item Writing) Basic Item Writing Principles: • One Correct Answer • Relevant and Appropriate • Realistic • Important to Related Task • Straightforward • Clear and Concise • No Clues that Give Away the Answer • Entry-Level Reading Level

  36. Developing the Exam Question(Item Writing) Item Writing Principles for Multiple-Choice: • Define the Task • Express a Complete Thought • Reduce the Reading Load • Use of Negative Words • Response Options need to: • Fits the Stem Logic • Equal in Length • Unique Meaning • Plausible • Avoid “All the Above” and “None of the Above”

  37. Put to the Test: Performance Analysis Nikki Shepherd Eatchel, M.A.

  38. Exam Question (Item) Performance Analysis There are various types of analyses used for evaluation of exam and item performance: • Item Response Theory (1 parameter, 3 parameter) • Classical Test Theory Classical Test Theory is the most common analysis type used for evaluation of state-based licensure exams.

  39. Exam Question (Item) Performance Analysis Classical Test Theory – Item Statistics p-value: Percentage of candidates who answer an item correctly rpbis: Correlation between performance on an item and performance on the overall exam

  40. Exam Question (Item) Performance Analysis Classical Test Theory – Item Statistics option analysis: Percentage of candidates who answer each option option rpbis: Correlation between performance on the options and the overall exam

  41. Exam Question (Item) Performance Analysis Classical Test Theory – Issues to Consider p-value: Too high, too low rpbis: Too low options: Distractors with no responses Distractors with more responses than key O rbpis: Distractors higher than key Omits Above expectations

  42. Exam Question (Item) Performance Analysis Classical Test Theory – Sample Statistics

  43. Exam Question (Item) Performance Analysis Classical Test Theory – Second sample

  44. Exam Question (Item) Performance Analysis Forensic Data Analysis – Item Level With the increasing security issues within the testing industry (e.g., brain dump sites, black market content, etc.) forensic data analysis is critical for the evaluation of a testing program.

  45. Exam Question (Item) Performance Analysis Forensic Data Analysis – Item Level What does Forensic Data tell you?

  46. Exam Question (Item) Performance Analysis Forensic Data Analysis Suspicious Candidate Activity • Candidates who did better on harder questions than they did on easier questions • Candidates who achieved a high score (in the top 20%) on the exam while only viewing questions for a very short period of time (over half of the items 10 seconds or less each)

  47. Exam Question (Item) Performance Analysis Forensic Data Analysis Suspicious Candidate Activity • Test takers who viewed some items for extended periods of time while cycling quickly (less than 8 seconds each) through the remaining items • Test takers with scores around chance-level while spending short amounts of time on each item (pre-viewing).

  48. Exam Question (Item) Performance Analysis Forensic Data Analysis – Item Level While there are a number of item and exam level statistics available for forensic data analysis, many of them revolve around classical test theory and item level performance. These types of analyses can be valuable in both test development forecasting and in evaluating security issues.

  49. The ARELLO Accreditation Program • Overview of the function and purpose

  50. The ARELLO Accreditation Program • Independent Evaluation • Confidence and Protection • Defensible Standards • Enhanced Quality • Facilitates License Recognition

More Related