Reflections and illustrations on DIF

1. Reflections and illustrations on DIF Paul De BoeckK.U.Leuven 25th IRT workshop Twente, October 2009

2. Reflections and illustrations on DIF Paul De BoeckUniversity of Amsterdam 25th IRT workshop Twente, October 2009

3. Is DIF a dead topic?A non-explanatory approach Paul De BoeckUniversity of Amsterdam 25th IRT workshop Twente, October 2009

4. Is there life after death for DIF?A non-explanatory approach Paul De BoeckK.U.Leuven 25th IRT workshop Twente, October 2009

5. The three DIF generations Zumbo, Language Assessment Quarterly, 2007 1st generation:from “item bias” to “differential item functioning” 2nd generation:modeling item responses, IRT, multidimensional models 3rd generation:explanation of DIF The end of history “ .. the pronouncements I hear from some quarters that psychometric and statistical research on DIF is dead or near dying ..”

6. Outline • Issues • Reflections and more • Possible answers

7. Issues • Anchoring • Statistic • Indeterminacies

8. I apologize, .. • There are already so many methodsyes • The best among the existing methodsare very good methodsyes • They are standard and good practiceyes • Do we really need more?no, therefore no real issues • And still

9. 1. Anchoring • Blind, iterativePurification- all other in step 1- nonrejected items in following steps • A priori set, test They workbased on pragmatism and a heuristic,on prior theory, what can one want more?

10. 2. Statistic and its distribution Based on difference per item or set of items • MH statistic • ST-p-DIF • Bu from SIBTEST • LR test statistic • Raju distance Other • Parameter estimates They work, what can one want more?

11. 3. Indeterminacies with an IRT modeling approach Basic model is • 1PL or Rasch modelfor uniform DIF • 2PLfor uniform and non-uniform DIF type 1 • 2PL multidimensionalfor uniform and non-uniform DIF type 2

12. Difficulties – uniform DIFAdditive or translational indeterminacyβfi = βri + δβiβ*fi = βri + δ*βi δ*βi = δβi + cβγ* = γ – cββfi , βri focal group and reference group difficultiesδβi DIF effectγ group effect * transformed values

13. Invariance of DIF explanation • δβi = Σk=0ωkXik (+ εi)Xik: value of item i on item covariate kωk: weight of covariate k in explaining DIF k=0 for intercept • ωk>0 are translation invariant, and only these covariates have explanatory value

14. Degrees of discriminationnon-uniform DIF type 1Multiplicative indeterminacy αfi = αri x δαi α*fi = αri x δ*αi δ*αi = δαi x cασθf* = σθf/ cαadditive formulation discrimination DIF

15. Loadings for multidimensional models The indeterminacies look a little embarrassing, because the results depend on one’s choice.

16. Reflections • Random item effects • Item mixture models • Robust statistics

17. Intro: Beliefs • DIF is gradualwhy not a random item effect? • DIF or no DIFwhy not a latent class of DIF items? • DIF items are a minoritywhy not identify outliers?

18. Where is the DIF?

19. Where is the DIF?

20. Where is the DIF?

21. Where is the DIF?

22. Intro: ANOVA approach • ηgpi = ln(Pr(Ygpi=1)/Pr(Ygpi=0)) • ηgpi = μoverall mean+ λgp = αθgp person effect, ability θgp ~ N (0,1)+ λi = βiitem effect, overall item difficulty+ λg = γg group effect+ λgp interaction p x g does not exist+ λgpi = α’iθgp interaction pwg x i+ λgi = β’gi interaction i x g uniform DIF+ λgpi = α’’giθgp interaction pwg x i x gnon-uniform DIF type 1 2PL version

23. + λgpi = α’iθgp+ λgpi = α’’giθgp interaction pwg x i x g isnon-uniform DIF Type 1 • + λgpi = α’iθgp1+ λgpi = α’’giθgp2 interaction pwg x i x g isnon-uniform DIF Type 2

24. Secondary dimension DIF g = 0 reference groupg = 1 focal group • ηgpi = (αi + gδαi)θgp + (βi + gδβi)+ λg = αiθgp +gδαiθgp+ (βi + gδβi)+ λg • Secondary-dimension DIF ηgpi = αiθgp1 + gδαiθgp2+ (βi + gδβi)+ λg Cho, De Boeck & Wilson, NCME 2009

25. can explain uniform DIF ηgpi = αiθgp1 + gδαiθgp2 + (βi + gδβi)+ λg gδαiμθg2 + gδαiθ’gp2 = gδβi Cho, De Boeck & Wilson, NCME 2009

26. Different from the MIMIC model • Secondary dimension DIF θgp1 ηgpi G θgp2 θgp1 ηgpi G gθgp2

27. 1. Random item effects • Within group random item effects(βri, βfi) ~ N(μβr, 0, σ2βr, σ2βf, ρβrβf)(βi, βf-gi) ~ N(μβr, 0, σ2β, σ2βf-g, ρββf-g)small number of parameters² • Idea based on Longford et al in Holland and Wainer (1993) for the MHthere is evidence that the true DIF parameters are distributed continuously Van den Noortgate & De Boeck, JEBS, 2005Gonzalez, De Boeck & Tuerlinckx, Psychological Methods, 2008De Boeck, Psychometrika, 2008

28. 2. Latent class of DIF items • Asymmetric DIF is exported to other items • Is avoided when DIF items are removed, appropriate removing eliminates interaction • Basis of purification process • Let us make a latent class for items to be removed, and identify the DIF items on the basis of their posterior probability

29. Item mixture model • ηgpi|ci=0 = θgp + βi non-DIF classηgpi|ci=1 = θgp + βgi DIF class non-DIF DIF reference θrp + βi θrp + β0i focal θfp + βi θfp + β1i Frederickx, Tuerlinckx, De Boeck & Magis, resubmitted 2009

30. further model specifications:- item effects are random- normal for the non-DIF items- bivariate normal for the DIF item difficulties- group specific normals for abilities

31. Simulation study 1PLP=500, 1000 2 I = 20, 50 x 2#DIF = 0, 5 (1.5, 1, 0.5, -1, -1.5) x 2 μθ1 = 0, μθ2 = 0, 0.5, x 2 = 16 μβ = μβ0 = μβ1, σ2β = σ2β0 = σ2β1 = 1, ρβ0β1 = 0five replicationsMCMC WinBUGS prior β variance: Inv Gamma, Half normal, Uniformdistributional parameters are estimatedposterior prob determines whether flagged as DIF

32. Results simulation studyaverage #errorsLRT 1.64MH 1.39ST-p-DIF 0.65mixture inverse gamma 0.30mixture normal 0.36mixture uniform 0.40item mixture does better or equally good then every other traditional method in all 16 cells

33. More results- results of mixture model are not affected by DIF being asymetrical- neither by true distribution of item difficulties (normal vs uniform)

34. 3. DIF items are outliers • Outlying with respect to the item difficulty difference between reference and focal group • Types of difference:- simple difference- standardized – divided by standard error- Raju distance – first equal mean difficulty linking, then standardizeτi = I/(I-1)2 x (di -d.)2/s2d is beta (0.5, (I-2)/2) distributed if di is normally distributed

35. Go robust:d. is replaced by the mediansd is replaced by mean absolute deviation Taking advantage of the fact that interitem variation is an approximation of se if robustly estimated De Boeck, Psychometrika 2008Magis & De Boeck, 2009, rejected

36. 20 items, nrs 19 and 20 are the true DIF items

38. Simple difference

39. Simple difference

40. Standardized difference

41. Standardized difference

42. Raju

43. Raju

44. Simulation study 1PLP=500, 1000 2 I = 20, 40 x 2%DIF = 0%, 10%, 20% x 3 size of DIF = 0.2, 0.4, 0.6, 0.8, 1.0 x 5μθ1 = 0, μθ2 = 0, 1 x 2 = 120100 replications

45. Results0% DIFMHSIBTESTLogisticRaju classicRaju robust Type 1 errors ≈ 5%

46. ResultsDIF size = 1, P=1000, I=40, equal μθ10% DIF20%DIFType 1 Power Type 1 PowerMH 0.10 1.00 0.23 1.00SIBTEST 0.10 0.98 0.21 0.97Logistic 0.10 1.00 0.20 1.00Raju classic 0.00 0.93 0.00 0.41Raju robust 0.04 1.00 0.02 1.00

47. Results are similar for unequal mean abilities • Results are similar but less pronouncedfor smaller P and smaller DIF size

48. Possible answers • Anchoring?Anchor set memberschip is binary latent item variable, or, the clean set of items • Statistic?Robust statisticworks also for nonparametric approaches • Indeterminacy?(go explanatory)no issue for random item model, look at the covequal means in item mixture approachequal means for Raju distance

49. Item mixtures and robust statistics do in one step what purification does in several steps, item by item, and through different purification steps – purification is approximate: • They both give a rationale for the solving the indeterminacy issue • Random item effect approach is not sensitive to indetermincay

50. Si no è utile è ben ispirazione Good for other purposes or a broader concept than DIF, for qualitative differences between groups • Random item models • Item mixture models • Robust statistics IRT