Download
validity and item analysis n.
Skip this Video
Loading SlideShow in 5 Seconds..
Validity and Item Analysis PowerPoint Presentation
Download Presentation
Validity and Item Analysis

Validity and Item Analysis

198 Vues Download Presentation
Télécharger la présentation

Validity and Item Analysis

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Validity and Item Analysis Chapter 4

  2. Validity • Concerns what the instrument measures and how well it does that task • Not something an instrument has or does not have • Informs counselor when it is appropriate to use the instrument and what can be inferred from the results • Reliability is a prerequisite for validity

  3. Movement Away from Traditional Categories of Validity • Traditional types of validity • Content-Related • Criterion-Related • Construct-Related

  4. Evidence Based on Instrument Content • The degree to which the evidence indicates the items, questions, or tasks adequately represent the intended behavior domain. • Central focus is typically on how the instrument’s content was determined • Content-related validation evidence should not be confused with face validity

  5. Evidence Based on the Response Processes • Information processing • Examining the response processes used by different groups • Computer simulated performances

  6. Evidence Based on Internal Structure • Examining internal structure using factor analysis • Examining the internal structure of an assessment with groups (e.g., gender, race or ethnicity)

  7. Evidence Based on Relations to Other Variables • Correlational method • Convergent evidence/Discriminant evidence • Multitrait-multimethod matrix • Prediction or instrument-criterion relationship • Concurrent and predictive validity • Regression • Decision Theory • Validity Generalization

  8. Correlational Method • Correlational method-examine relationship between instrument and other pertinent variables or a criterion • Validity coefficient(s)

  9. Prediction or instrument-criterion relationship • Concurrent validity • Predictive validity

  10. Regression . . 5 . . … … . 4 . . . … Ratings … . … . . . 3 . .. . . .. .. . .. 2 . 1 10 20 30 40 50 60 70 80 90 100 ABC

  11. Regression Line . . 5 . . … … . 4 . . … . Ratings … . Y' = a + bX … . . . 3 . .. . . .. .. . a = intercept b = slope X = score .. 2 . 1 10 20 30 40 50 60 70 80 90 100 ABC

  12. The margin of expected error in the individual’s predicted criterion score as a result of imperfect validity. Standard Error of Estimate

  13. Decision Theory: Expectancy Table ABC Scale Ratings

  14. Expectancy Table (cont.) Misses Hits Hits Misses

  15. Expectancy Table (cont.) False Negatives False Positives

  16. Validity Generalization • A method of combining validation studies to determine if the validity evidence can be generalized • Must be a substantial number of studies • Meta-analysis

  17. Evidence-based consequences of testing • Social consequences of racial/ethnic group differences • Consider social implications when examining the validation evidence

  18. Conclusion on validation evidence • Gradual accumulation of evidence • Counselor must evaluate the information to determine if it appropriate for which client under what circumstance • Validation evidence should also be considered in informal assessments

  19. Item Analysis • Focus is on examining and evaluating each item on an instrument. • Item difficulty: p = # who answered correctly total number • Item discrimination: d = upper % - lower % (or correlational method)

  20. Item Response Theory or Latent Trait Theory • Focus is on each item and on establishing items that measure the individual’s ability or level of a latent trait. • Involves examining the item characteristic function (or characteristic curve) and the calibration of each individual item. • Parameters: • Difficulty • Slope • Basal level

  21. Item Characteristic Curve 1.00 .50 .00 % 4 5 6 7 8 9 Grade