Conceptual Issues in Response-Time Modeling

1. Conceptual Issues in Response-Time Modeling Wim J. van der Linden CTB/McGraw-Hill

2. Outline • Traditions of RT modeling • RTs fixed or random? • Item completion, responses, and RTs • RT and speed • Speed and ability

3. Outline Cont’d • RT and item difficulty • Dependences between responses and RTs • Hierarchical model of responses and RTs • Applications to testing problems

4. Traditions of RT Modeling • Four different traditions • No model • Distinct models for RTs • Response models with RT parameters • RT models in mathematical psychology • Alternative • Hierarchical model of responses and RTs • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

5. Fixed or Random RTs • Some models treat RTs as fixed quantities: • Roskam (1987, 1997); Thurstone (1937) • RTs treated as random in psychology • Random responses but fixed RTs seems contradictory • Conclusion 1: Just as responses, RTs ontest items should be treated as realizationsof random variables • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

6. Item Completion, Response,and RT • Rasch (1960) models for misreadings andreading speed • Poisson-gamma framework • Same notation and terminology for parameters in both types of models • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more “To which extent the two difficulty parameters … and the two ability parameters … run parallel is a question to be answered by empirical results, and at present we shall leave it open.” (Rasch, 1960, p. 42)

7. Item Completion, Response,and RTCont’d • Notion of equivalent scores of speed tests (Gulliksen, 1960; Woodbury (1951, 1963): • Total time on a fixed number of items • Number of items correct in a fixed time interval • Three types of variables required to describetest behavior: • Tij: response time (person j and item i) • Uij: response • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

8. Item Completion, Response,and RTCont’d • Three sets of variables (cont’d) • Dij: item completion (design variable) • Uij and Dij have different distributions • Same holds for their sums • NU: number-correct scores • ND: number of items completed • Equivalence only when Pr{Uij=1|Dij=1}=1for all items and persons • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

9. Item Completion, Response,and RTCont’d • Distinction between speed and power test makesno sense; all test are hybrids • Conclusion 2: Tij, Uij, and Dij are randomvariables with different distributions. The same holds for their sums: total time (T), numbercorrect (NU), and number completed (ND). Except for discreteness, T and ND are inversely related. (We’ll assume T and NU to beindependent!) • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

10. RT and Speed • Speed and time are no equivalent notions • Generally, speed is a rate of change of some measure with respect to time, e.g., • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

11. RT and Speed Cont’d • For achievement testing, an appropriate notion of speed is cognitive speed: • Fundamental equation: Amount of labor required (“time intensity”) by item i Speed of person j Response time of person j on item i

12. RT and Speed Cont’d • Lognormal RT model (van der Linden, 2006) • Log transformation to remove skewness from RT distributions • Addition of random term

13. Speed Time intensity Discrimination RT and Speed Cont’d • Lognormal RT model:

14. RT and Speed Cont’d • Conclusion 3: RT and speed are different concepts related through a fundamental equation. RT models with a speed parameter should also have an item parameter for their amount of cognitive labor (or time intensity)

15. Speed and Ability • Speed-accuracy tradeoff in psychology is same as a speed-ability tradeoff in achievement testing • Negative within-person correlation between τ and θ • Change of speed required for tradeoff to become manifest • Traditional IRT view of a person’s ability is of θas a scale point, not as a function θ=θ(τ) • Effective ability level • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

16. Speed and Ability Cont’d • At group level, any correlation between ability and speed may occur • Basic assumption: constancy of speed duringthe test • Constant speed implies constant ability (ceterisparibus) • In practice, speed and ability always fluctuate somewhat, but fluctuations should be minorand unsystematic • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

17. Speed and Ability Cont’d • Conclusion 4: Speed and ability are related through a distinct function θ=θ(τ) for each test taker. The function itself need not be corporated into the response and RT models. But these models do require (fixed) parameters for the effective ability and speed of the test takers. • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

18. RT and Item Difficulty • Descriptive research and speed-accuracy tradeoff suggest correlation between RT and item difficult • Item difficulty parameter in RT model? • Counterexample • Item parameters in response and RT models are for different item effects (on probability of correct response and time, respectively) • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

19. RT and Item Difficulty Cont’d • Latent vs. manifest effect parameters • Danger of reification of latent effects • Conclusion 5: RT models require item parameters for their time intensity but difficulty parameters belong in response models • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

20. Dependences between Responses and RTs • Descriptive vs. experimental studies • However, these studies necessarily involvedata aggregation across items and/or persons • Spurious correlations due to hidden sources of covariation (item and person parameters) • Marginal vs. conditional independence between responses (spurious correlation, Simpson’s paradox, etc.) • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

21. Dependences between Responses and RTs Cont’d • Conclusion 6: Regular test behavior is characterized by three different types of conditional (or “local”) independence,namely between • responses on different items • between RTs on different items • between responses and RTs on the same item • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

22. Dependences between Responses and RTs Cont’d • For these conditional independencies to holdfor an entire test, constant speed is a necessary condition • Empirical results • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

23. Hierarchical Model ofResponses and RTs • Distinct models for responses and RTs for a fixed person and item • Regular IRT model • E.g., lognormal model for RTs • Models should have • parameters for effective ability and speed • parameters for item difficulty and time intensity • conditional independence • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

24. Hierarchical Model ofResponses and RTs Cont’d • Second-level models for dependences between • ability and speed across persons • difficulty and time intensity across items • Multivariate normal distributions (possibly after parameter transformation) • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

25. Hierarchical Model ofResponses and RTs Cont’d • Bayesian treatment of modeling framework • Parameter estimation and model fit analysis with MCMC (Gibbs sampler) • Plug-and-play approach • Calibration of items with respect to RT parameters is straightforward • R package available upon request (Fox, Klein Entink, & van der Linden, 2007; Klein Entink,Fox, & van der Linden, 2009) • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

26. Applications to Testing Problems • Test design • Adaptive testing • Item selection • Differential speededness • Detection of cheating • Item memorization and preknowledge • Collusion • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

27. Applications to Testing Problems • Use of RTs as collateral information in parameter estimation • Cognitive research on problem solving • Etc. • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

28. No RT Model • Descriptive studies in educational testing • Correlation between responses and RTs • Regression of RT on item and person attributes • Word counts, IRT item parameters, etc. • Number-correct scores; ability estimates • Experimental studies in psychology • Manipulation of task or conditions • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

29. t No RT Model Cont’d • Experimental reaction-time research (cont’d) • Speed-accuracy tradeoff (Luce, 1986) • Plot of proportion of correct responses against RT • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

30. No RT Model Cont’d • Problems • Spurious correlations between observed RTs • Speed-accuracy tradeoff is not a between-person phenomenon • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

31. Stroop Test Green

32. Stroop Test Cont’d Blue

33. Spurious Relations • RTs of two arbitrary students on a quantitative reasoning test • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more Subject 1: 22, 19, 40, 43, 27, 27, 45, 23, 14, … Subject 2: 26, 38, 101, 57, 37, 21, 116, 44, 10, …

34. Spurious RelationsCont’d • RTs of two arbitrary students on a quantitative reasoning test • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more Subject 1: 22, 19, 40, 43, 27, 27, 45, 23, 14, … Subject 2: 26, 38, 101, 57, 37, 21, 116, 44, 10, … r= .89

35. Spurious RelationsCont’d • RTs of two arbitrary students on a quantitative reasoning test • Responses of same students • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more Subject 1: 22, 19, 40, 43, 27, 27, 45, 23, 14, … Subject 2: 26, 38, 101, 57, 37, 21, 116, 44, 10, … r= .89 Subject 1: 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, … Subject 2: 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1,… r= .20

36. Spurious RelationsCont’d • RTs of two arbitrary students on a quantitative reasoning test • Responses of same students • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more Subject 1: 22, 19, 40, 43, 27, 27, 45, 23, 14, … Subject 2: 26, 38, 101, 57, 37, 21, 116, 44, 10, … r= .21 Subject 1: 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, … Subject 2: 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1,…

37. Distinct Models for RTs • Rasch’s (1960) models for reading speed • Exponential models • Oosterloo (1975); Scheiblechner (1979) • Gamma models • Maris (1993); Pieters & van der Ven (1982) • Weibull models • Tatsuoka & Tatsuoka (1980)

38. Rasch’s Models • Poisson distribution of number of readingerrors a in a text of N words • Gamma distribution of reading time for textof N words

39. Response Models with RT Parameters • This type of model mostly motivated by attempts to build speed-accuracy tradeoff in response model • Response surface in Thurstone (1937) • Logistic models • Roskam (1987; 1997); Verhelst, Verstralen &Janssen (1997) • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

40. Response Models with RT Parameters • We also have RT models that incorporate response parameters • E.g., lognormal models by Gaviria (2005) and Thissen (1982) • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

41. Thurstone’s Response Surface

42. Roskam’s Model (1997) RT Ability “Speed-accuracy tradeoff” Item difficulty

43. RT Models in Mathematical Psychology • Models for underlying psychological processes • Diffusion models • Models for sequential and parallel processing • Experimental data • Standardized task • Assumption of exchangeable subjects • No subject or item parameters • Test design • Fixed tests • Adaptive tests • Test accommodations • And many more

44. RT and Speed Item 2 Item 1 375 229 58 39 375 229 + + Time: 9 sec 12 sec Speed: ? ?

45. Ability Within-person relation Speed Speed-Ability Tradeoff

46. Ability Higher ability Lower ability Speed Speed-Ability Tradeoff Cont’d

47. Effective ability θ=θ(τ) Speed τ Effective speed Speed-Ability Tradeoff Cont’d

48. Speed-Ability Tradeoff Cont’d x x Ability x x x x x x x Speed

49. Speed-Ability Tradeoff Cont’d x x x Ability x x x x x x Speed

50. Speed-Ability Tradeoff Cont’d x x Ability x x x x x x x Speed