Eye tracking

1. Eye tracking Applications within cognitive science Dr. Christa van Mierlo

2. Why is eye tracking used in cognitive science? Scan patterns give delayed information about the mental processes that are developing in a person’s mind and reveal what visual information is (going to be) used by these processes.

3. Frequently studied EM components • Fixations • Gaze stays fixed on one position • Intake of new visual information • Planning of new eye movement • Saccades • Fast movement of both eyes in the same direction • Processing of new visual input is limited: low spatial frequencies attentuated high spatial frequencies unaffected. • Top velocity proportional to amplitude

4. Visual Search • The sometimes difficult process of finding a target among distractors in often cluttered visual environments. • Physical and cognitive processing limitations can prevent us from instantly recognizing the presence of a target item in a single glance (e.g. a large number of shared features with distractor or fuzzy target specifications) • This can be overcome by focusing attention: • Bottom up • Top down

5. Bottom-up factors drawing attention • onsets (e.g., Theeuwes, Kramer, Hahn, Irwin, & Zelinsky, 1999; Yantis & Jonides, 1984) • unique colors (e.g., Theeuwes, 1994; Theeuwes & Burger, 1998) Even when the location of the target is known, highly salient features that are known not to be associated with the target can still capture attention (Christ & Abrams, 2006).

6. Top-down factors focusing attention • Target specificity: the number of features the candidate shares with the target (e.g., Folk, Remington, & Johnston, 1992; Folk, Remington, & Wright, 1994) • Memory (Boot, McCarley, Kramer, & Peterson, 2004; Brockmole & Henderson, 2006; Peterson & Kramer, 2001).

7. Four Eye Tracking studies within Visual Search • The effects of target template specificity on visual search in real-world scenes: Evidence from eye movements (Malcolm & Henderson, 2009) • Comparing eye movements to detected vs. undetected target stimuli in an identity search task (Jacob & Hochstein 2009) • Stable individual differences in search strategy?: The effects of task demands and motivational factors on scannning strategy in visual search (Boot et al., 2009) • Where to look next? Eye movements reduce local uncertainty (Renninger et al. 2007)

8. The effects of target template specificity on visual search in real-world scenesMalcolm & Henderson (JOV 2009) Searching is faster for more explicitly specified targets. Why? • Does this affect the activation map that is used to select probable target regions for fixation? • Does this allow for faster evaluation of a target candidate? • Or does it simply allow the search to begin faster?

9. Method Subjects had to look for a specific object within a visual scene. • Target could be specified by a word or a picture. Pictures specify the target template more elaborately than words. • To manipulate the time that the subject had to build up a target template and keep it salient in memory, they manipulated the SOA between the cue presentation and the onset of visual scene (short/long). • To manipulate target familiarity, the target specification was either shown 4 times to the subject prior to experiment or not at all.

10. Analysis Divide scanpaths in different epochs: • timing of first saccade (= time it takes to determine the first possible candidate for target) • time it takes to find target (all saccades and fixations up to the first fixation on the target, representing processes in which target candidates are selected and rejected) • time it takes to decide that the object really is the target (verification)

12. Results Picture rather than word cues resulted in: • Faster total search times • Shorter scanning and verification times • Fewer regions visited • Shorter scanning fixation durations (rejection of distractors) Longer SOA’s resulted in faster search initiation, but no interaction with cue type

13. Discussion Knowledge of a target’s appearance prior to search benefits scanning in 2 ways: • Facilitating the selection of potential target locations • Decreasing the time that it takes to reject fixated distractors before moving on to next potential target

14. Proposed neural mechanism • People represent the visual scene in an activity map • Search is accelerated by increasing the topographical activity that is associated with target similar features and decreasing the noisy activity of target irrelevant features. So that candidate selection and verification of the target happen quicker.

15. Comparing eye movements to detected vs. undetected target stimuli in an identity search taskJacob & Hochstein JOV 2009 In conscious search: • What determines of a target will be found? • Does conscious detection come before or after concentrated fixations on the target? • What is the relation between repeated fixations on the same scene region and limited WM capacity? • What in the sequence of fixations reflects or influences ultimate conscious perception?

16. Method • Find two identical cards among distracters • In each set there were two pairs of identical cards instead of just one • Participants were not informed of this • After a learning session of 100 trials their eye movements were measured for 50 trials

18. Analysis Compare fixations on detected targets with fixations on undetected targets the detected pair in red the undetected pair in blue.

24. Results • More and longer fixations on detected items than on undetected items. • Less distance between fixations on detected than on undetected items. • The patterns of fixations are nearly identical up to the point of approximately 4 fixations before the end of the trial (~1.5 s before the first mouse click). This is true for both long and short trials. • The number of fixations needed for identification is more or less fixed within a range.

25. Discussion So fixations are needed to identify the target; detection is not an inherent property of the stimulus! Does the large number of fixations give rise to detection or is it a result of detection?

26. The fixations just before the mouse click did not always land on the target cards, indicating that they were not the results of an verification process. • The relatively small increase in cumulative number of fixations on detected pairs in long searches, implies that the number of fixations on targets needed for identification is defined within a certain period of time. So it seems that the increase in fixations near the target are necessary for its detection and do not result from verification!

27. Why a short burst of fixations near target cards just before detection? It may be difficult to keep many cards in working memory at the same time, so that fixations need to be close to each other to associate place with identity. Since the sequential distance decreases when approaching detection; perhaps a necessary condition for detection is that two cards be represented concurrently in working memory.

28. Detection may depend on the increase in fixations rising above some threshold. • The point where the slope exceeds a pre-determined threshold may be regarded as the bifurcation point where there is a change of state in the search process; a transition between a first stage of “search in the dark” to a second stage of “early implicit recognition”.

29. Proposed model • Stage 1: Initial search; random fixations on the different cards in arbitrary order. • Stage 2: Implicit (unconscious) recognition of the target pair, perhaps controlling and guiding eye movements to the relevant sensed location of these target cards. • Stage 3: Insight: Explicit detection with conscious knowledge of target presence and its location followed by rapid marking of the two cards.

30. Stable individual differences in search strategy? The effects of task demands and motivational factors on scannning strategy in visual searchBoot et al., JOV 2009 This study seeks to further evaluate and understand individual differences in visual search behaviour in the context of search tasks in which poor strategies can have a major impact on performance.

31. Background • In Boot et al. (2006) , participants viewed dynamic displays in which up to 24 dots moved across the display. • During some trials a new dot appeared in the display and the task of the participant was to push a button when this occurred.

32. A surprisingly large range in accuracy: some participants almost always detected the new dot others missed 50% or more of the onset events

33. The more participants moved their eyes among moving objects in the display, the fewer targets they detected. • When overt searchers were instructed to search covertly, their performance matched the performance of covert searchers. Conversely, covert searchers instructed to search overtly performed just as poorly as overt searchers. • This ability to switch strategies suggests that strategy is not dictated by the size of an individual’s attentional field or individual differences in visual processing.

34. Their current study seeks to explore: • whether stable individual differences in preference for a certain scanning strategy might explain maladaptive scan strategies • the degree to which strategy might be modulated by task demands, feedback, motivation and monetary incentives

35. Method Study scanning strategies during: • dynamic dot detection task • an efficient search task (a 45° left or right tilted line among vertical lines) • an inefficient search task (a tilted T among randomly tilted Ls that had an offset of the _) • a change blindness task in which participants searched for changes in driving scenes (change in colour, presence or position masked by other changes).

36. Change blindness and inefficient search require focal attention to the target (overt attention). • Dynamic dot detection task and efficient search task do not (covert attention).

37. Analysis • As a measure of overt versus covert searchers: average number of eye movements made per second. • averaged across set sizes. • correlated across tasks: If participants use the same scan strategy in different tasks, regardless of whether or not this strategy is adaptive, then the rates of eye movements on the different search tasks should be correlated.

38. Predictions Performance in the dynamic dot detection task has been shown to be almost exclusively driven by strategy (Boot et al., 2006). If the eye movements on this task differ individually but are similar to that those seen on other visual search tasks for each subject, these differences in scan pattern are likely to be caused by differences in strategy choice, not differences in visual processing ability.

39. In difficult and inefficient search tasks, a covert search strategy would be highly maladaptive due to the difficulty of discriminating complex stimuli in the periphery. In an efficient or easy search task, eye movements might hinder performance by focusing attention on individual items rather than allowing the unique target item to pop-out.

40. Results • A covert scanning strategy was the most optimal strategy in the dynamic dot detection task • A clear trend toward faster response times was found for more overt searchers in the change blindness task. • An overt scanning strategy was the most optimal strategy for the inefficient search task • No effect of scanning strategy on performance for the efficient search task

44. Observers retain their scanning strategy across different tasks; however, they also adjust their scanning strategy depending on the task performed. • Those observers who adjust their scanning strategy to a greater degree exhibit the greatest overall benefit in accuracy.

45. Discussion • Although strategy remained similar, task-specific modulation of saccade rate was clearly observed. Participants made fewer saccades in tasks such as the dynamic dot detection task and the efficient search task compared to the change blindness task and the inefficient search task. • However, in general, strategy tended to remain similar across tasks, even when that strategy resulted in slow or inaccurate performance.

46. Experiment 2 • Can participants modify their scanning approach when it becomes clear that it is resulting in poor performance? • In Experiment 2, participants were provided with feedback after each trial, and monetary incentive to ensure feedback would be attended. • If participants do not modify their strategy this would be evidence of strong, stable individual differences. • If participants change their strategy based on feedback and motivation, similar strategy across many tasks seems to be a weak preference to utilize one strategy over another under conditions of uncertain performance and low consequences.

47. Method • Dynamic dot detection versus inefficient search (a tilted T among randomly tilted Ls that had an offset of the _ of the Ls). • Explicit feedback • for DDD: ‘incorrect; you missed the target/no target present’ or ‘correct – target/no target’ present’ • For IS: ‘You were fast!’ or ‘A bit slow!’ Fastest participant received an additional 20 dollars in payment

49. Results • Feedback and monetary incentives caused participants to shift their strategy rather than maintain similar strategies across tasks. • Thus, based on situational factors, participants will abandon their default strategy and adopt a strategy that is more adaptive to the task at hand.

50. General Discussion • Scan strategies remain stable across a variety of both static and dynamic tasks when the relationship between strategy and performance is unclear or motivation to perform well is low. • This suggests that participants might be utilizing a default strategy.