Analyzing the Integration of Figures and Text Using Eye Movements

1. Analyzing the Integration of Figures and Text Using Eye Movements Jason Rubinstein With Eileen Kowler, Nick Ross, Elio Santos, Min Zhao University of Delaware and Rutgers

2. Introduction The L–D differences in the ventilatory and metabolic variables occurred even between 15-min epochs of identical activity levels, whether these were below or above the daily mean (Fig. 5). VE/VO2 in the D remained higher than in the L, by 13% when activity was low and 9% at the higher activity level. The results of this analysis did not depend on the animal ’s activity immediately preceding the epoch analyzed, as there was no correlation between the L –D difference in VE for iso-activity epochs and the level of activity prior to the epochs analyzed. • Figures + Text • Same “page” vs. different “page” • Semantic integration of information over long timescales • Building up a knowledgebase • Between saccades • “rapid movements of the eyes between positions of rest” • Between “page turns” • Meaning • People tend to limit memory load when possible

3. Ballard, Hayhoe, & Pelz, 1995

4. Epelboim & Suppes, 2001

5. Question What strategies do people use to extract meaning from figures and text (in combination)?

6. Why? • Real world • Textbooks • Educational material • Standardized tests • SATs • Newspapers • Websites • Instruction manuals • Captions

9. The percentage statements during which the participant looked at the person was used as an indicator of gaze seeking. There was not an effect of gaze cueing and no interaction between gaze cueing and full or partial instructions. For full instructions, the person was fixated less often when he provided gaze cues, than when he did not provide gaze cues. For partial instructions, gaze cues were often sought.

11. The percentage statements during which the participant looked at the person was used as an indicator of gaze seeking. There was not an effect of gaze cueing and no interaction between gaze cueing and full or partial instructions. For full instructions, the person was fixated less often when he provided gaze cues, than when he did not provide gaze cues. For partial instructions, gaze cues were often sought.

12. Which of the following is true? A. The percentage of fixations on the person was high for both full and partial instructions independently of whether the gaze cues were provided by the person or not. B. Fixations on the person were more frequent following full instructions with or without gaze cues. C. Gaze cues increased the percentage of fixations on the person for both full and partial instructions. D. For partial instructions, fixations on the person were rare whether gaze cues were provided or not.  

13. Question • What strategies do people use to extract meaning from figures and text (in combination)? • Patterns in • Rule-governed transitions • Revisits • “Effector” (motor action) • Eye (simultaneous) or arm (sequential) vs

15. VIDEO

16. Results • Revisit figure after reading key words in text • Words that refer to the legend, specific aspects of figure, etc

17. Results • Revisit figure after reading key words in text • Words that refer to the legend, specific aspects of figure, etc • On average… • Switch more in simultaneous condition (!)

18. Number of visits to text Number of visits to figure

19. Results • Revisit figure after reading key words in text • Words that refer to the legend, specific aspects of figure, etc • On average… • Switch more in simultaneous condition (!) • Though “easier” (cognitively) in alternating • Less “work”

20. Results • Revisit figure after reading key words in text • Words that refer to the legend, specific aspects of figure, etc • On average… • Switch more in simultaneous condition (!) • Though “easier” (cognitively) in alternating • Less “work” • Periphery -- planning

21. Results • Revisit figure after reading key words in text • Words that refer to the legend, specific aspects of figure, etc • On average… • Switch more in simultaneous condition (!) • Though “easier” cognitively in alternating • Less “work” • Periphery -- planning • Spend more time in text in both conditions • Longer visits + more time overall

22. Proportion of time spent in figure Proportion of time spent in text

23. Results • Revisit figure after reading key words in text • Words that refer to the legend, specific aspects of figure, etc • On average… • Switch more in simultaneous condition (!) • Though “easier” cognitively in alternating • Less “work” • Periphery -- planning • Spend more time in text in both conditions • Longer visits + more time overall • However, proportion of overall time between simultaneous and alternating about the same

24. Average length of visits to figure (s) Average length of visits to text (s)

25. Results • Revisit figure after reading key words in text • Words that refer to the legend,specific aspects of figure, etc • On average… • Switch more in simultaneous condition (!) • Though “easier” cognitively in alternating • Less “work” • Periphery -- planning • Spend more time in text in both conditions • Longer visits + more time overall • However, proportion of overall time between simultaneous and alternating about the same • Reading… • Images vs. words?

26. What next? • MORE • Participants • Stimuli • Shorter trials/harder questions • Accuracy of various strategies • Specifics of transitions • Word in text → which part of figure? • Vice versa • Model revisits • Simultaneous trials • Saccade triggers stimuli • Periphery

27. ?

28. Thank you! • Eileen Kowler • Nick Ross, Elio Santos, Min Zhao • Matthew Stone • Joe, Henry, Avery, Rachel, Lara • The Academy • Everyone else