Measuring Human Intelligence with Artificial Intelligence

1. Measuring Human Intelligence with Artificial Intelligence Adaptive Item Generation Susan E. Embretson Sangyoon Yi

2. Introduction • Adaptive Item Generation • Intelligence testing • Generate optimally informative item for the examinee during the test • Optimally informative item • Based on the previous pattern of the examinee’s response • Ex. Deep Blue (Chess Computer)

3. Introduction • Adaptive Item Generation • Psychometric methods for adaptive testing • Intelligence measurement • Adaptive item selection leads to shorter and more reliable tests • A cognitive analysis of items • Knowledge is required of how stimulus features in specific items impact the ability construct

4. Introduction • Adaptive Item Generation {f_1, f_3, f5} impact Psychometric properties

5. Cognitive Design System Approach to Adaptive Item Generation • Theoretical Foundations for Cognitive Design Systems • Supporting Developments • Stages in Applying Cognitive Design Systems • Supporting Data for Cognitive Design Systems • Initial Cognitive Model for Matrix Items • Algorithmic Item Generation and Reversed Cognitive Model • Item Generation by Artificial Intelligence • Empirical Tryout of Item Generation • Related Approaches to Item Development • Evolution of Approach : Advantage and Disadvantages • Future

6. Cognitive Design System Approach to Adaptive Item Generation • Matrix completion problems • Regard this item type as central to measuring intelligence

7. Cognitive Design System Approach to Adaptive Item Generation • Cognitive processing model for the item type • It measures the construct • For adaptive item generation • A conceptualization of construct validity • Psychometric models • A computer program

8. Cognitive Design System Approach to Adaptive Item Generation • Theoretical Foundations for Cognitive Design Systems • Based on an information processing theory of the item type. • Originated with cognitive component analysis of complex item types for measuring intelligence …

9. Cognitive Design System Approach to Adaptive Item Generation • Theoretical Foundations for Cognitive Design Systems • Cognitive theory • Specifies the impact of processes on performance, and the impact of stimulus features on processes Stimulus features Processes Performance

10. Cognitive Design System Approach to Adaptive Item Generation • Supporting Developments • Construct Validity and Cognitive Design Systems • Psychometric Models for Cognitive Design Systems • Computer programs for Adaptive Item Generation

11. Supporting Data for Cognitive Design Systems • Initial Cognitive Model for Matrix Items • Advanced Progressive Matrices(Raven, et al. 1992) • Algorithmic Item Generation and Reversed Cognitive Model

12. Supporting Data for Cognitive Design Systems • Item Generation by Artificial Intelligence • Ex) ITEMGEN1 • Randomly selects stimuli and their attributes to fulfill the structural specifications • Empirical Tryout of Item Generation • Item generation has not been attempted with the full cognitive approach for the matrix completion items

13. Related Approaches to Item Development • Traditional Approach • Item writing as an art • By human • Item model Approach • Items are “variablized” • Item parameters are invariant over the cloned items • Ex) an existing mathematics word problem

14. Evolution of Approach : Advantages and Disadvantages • Advantages • New items may be readily developed • Items may be developed to target difficulty levels and psychometric quality • New items may be placed in the item bank without empirical tryout • Construct validity is available at the item level • Tests may be redesigned to represent specifically targeted sources of item difficulty

15. Evolution of Approach : Advantages and Disadvantages • Disadvantages • The approach requires substantial initial effort • The approach works best for item types that already have been developed

16. Future • Item generation by artificial intelligence fulfills practical needs for new items • The many correlates and relationships of intelligence measurements to other variables may be understood more clearly if the characteristic processing at different ability levels can be explicated