Fit of Ideal-point and Dominance IRT Models to Simulated Data

1. Fit of Ideal-point and Dominance IRT Models to Simulated Data Chenwei Liao and Alan D Mead Illinois Institute of Technology

2. Outline • Background and Objective • Hypotheses and Methods • Results • Discussions

3. Background • Personality Used in personnel selection - Incremental validity to predict job performance beyond cognitive ability (Barrick & Mount, 1991; Ones et al, 1993) - Less adverse impact (Feingold, 1994; Hough, 1996; Ones et al, 1993). • Model-data-fit - Need to calibrate personality traits - Use IRT models - Degree of fit depends on data structure

4. Background (cont.) • Item response processes – thinking of data structure • IRT models and item response processes: 1) Traditional dominance IRT models: - high trait - high probability of endorsing 2) Ideal-point IRT models - similar item & trait – high probability of endorsing

5. Background (cont.) Dominance Model IRF: - x: Theta (trait level) - y: Probability of endorsing Ideal-point Model IRF: - x: distance between person trait and item extremity - y: Probability of endorsing

6. Background (cont.) • Chernyshenko et al, (2001) - Traditional dominance IRT models have failed. Suggest to look at item response processes and Ideal-point IRT models • Stark et al. (2006) - Ideal-point IRT models: as good or better fit to personality items than do dominance IRT models • Chernyshenko et al. (2007) - Ideal-point IRT method: more advantageous than dominance IRT and CTT in scale development in terms of model-data-fit

7. Limitation of previous studiesand objective of current study • Limitation of previous studies - Unknown item response processes! • Objective of current study 1) Investigate model-data-fit by utilizing simulation with known item response processes 2) Test the assumption that the best fit model represents data underlying structure of response processes

8. Current Study

9. Models • Dominance: - Samejima’s Graded Response Model (SGRM); • Ideal Point: - General Graded Unfolding Model (GGUM). • Larger sample and longer test were said to be related to a better fit (Hulin et al, 1982; De la Torre et al, 2006).

10. Hypotheses Generating models • H1: Data generated by an ideal point model will be best fit by an ideal-point model and data generated by a dominance model will be best fit by a dominance model. • H2: The ideal point model will fit the dominance data better than the dominance model will fit the ideal-point data. • H3: The ideal-point model will fit the mixture data better than the dominance model.

11. Hypotheses (cont.) Sample Sizes • H4: All models will fit better in larger samples. • H5: The GGUM model will fit relatively worse in smaller samples, as compared to simpler, dominance models. Test Lengths • H6: The GGUM model will fit relatively worse for very short tests, as compared to longer tests.

12. Datasets • Self-Control Scale from the 16PF • Procedure:1) Calibrate 16PF data to get item parameters - SGRM: PARSCALE4.1; GGUM: GGUM2004.2) Generate simulated data: - models: ideal point/dominance/mixed; - sample size: 300, 2000; - test length: 10, 37; - 50 replications;

13. Model-Data-Fit • Cross validation ratio: each item in each condition • Only singles – simulation study assures unidimensionality assumption • Smaller value – better fit • Frequencies of ratios were tallied into 6 groups: very small (<1), small (1-<2), medium (2-<3), moderately large (3-<4), large (4-<5), very large (>=5).

14. Results overview

15. Results

16. Discussion (1) • “GGUM fits better” - Confirm previous findings. - However, because regardless of the underlying response process, GGUM fits better than SGRM, it does not demonstrate that the response process or IRF/ORF is non-monotone. The previous assumption does not hold true. - Possible reason: Software (PARSCALE & GGUM) manifest models differently • Better fit in small samples, especially for SGRM - Explanation: chi-square is sensitive to sample size

17. Discussion (2) • Examine similarities of the theta metrics - Negative correlation between theta estimates from GGUM and those from SGRM

18. Discussion (3) GGUM: - Reverse the estimate - Add a constant in scaling Scaling issue

19. Thanks!