Background elements RT & Perceptual States with masked and not-masked stimuli

1. ON THE RELATIONSHIP BETWEEN MOTOR AND PERCEPTUAL BEHAVIOR – A SDT FRAMEWORKAndrei GoreawithPedro Cardoso-LeiteFlorian WaszakPascal MamassianLaboratoire de Psychologie de la PerceptionCNRS & René Descartes University71 Ave Edouard Vaillant, 92774 Boulogne-Billancourt, France

2. SYNOPSYS • Background elements • RT & Perceptual States with masked and not-masked stimuli • 1 A one-path – two-decisions model • RT & Temporal Order Judgments • 1 One-path – two-decisions model again • General conclusions

3. PART I SOME BACKGROUND

4. A. Layman’s view Action with perceptual awareness Sensory Input Perceptual Decision Verbal report aware / not aware

5. Priming & Metacontrast + RT • Fehrer, E. & Raab, D. (1962) • Harrison, K. & Fox, R. (1966) • Schiller, P. H. & Smith, M. C. (1966) • Neumann, O. (1982) • Neumann, O., Esselmann, U. & Klotz, W. (1993) • Klotz, W. & Neumann O. (1999) • Taylor, J.L. & McCloskey, D.I. (1990) • Steglich, C. & Neumann, O. (2000) • Schmidt, T. (2002) • Ogmen, H., Breitmeyer, B. & Melvin, R. (2003) • Scharlau, I. & Ansorge, U. (2003) • Scharlau, I. & Neumann, O. (2003) • Vorberg, D., Mattler, U., Heinecke, A., Schmidt, T. & Schwarzbach, J. (2003) • Rossetti, Y. & Pisella, L. (2002) • Breitmayer, B., Ro, T. & Singhal, N. S. (2004) TOJ & RT • Roufs, J.A.J. (1974) • Jaskowski, P. (1991, 1992, 1993, 1996) • Tappe, T., Niepel, M. & Neumann, O (1993) • Jaskowski, P. & Verleger, R. (2000) • Spence C., Baddeley R., Zampini M., James R. & Shore D.I. (2003) • Adams, W.J & Mamassian, P. (2004) Perception & Gaze pursuit • Beutter, B.B. & Stone, L.S. (1998) • Beutter, B.B. & Stone, L.S. (2000). • Gegenfurtner, K.R., Xing, D., Scott, B.H. & Hawken, M.J. (2003) • Osborne, L.C., Lisberger, S.G. & Bialek, B. (2005) • Morrone, M.C., Raffele, S., Ma-Wyatt, A. & Ross, J. (2005) Most frequently used experimental paradigms

6. Lateral Interactions (implicit) B. The 2 pathways view Decision rule not specified Action with or without perceptual awareness ? dorsal Sensory Input ventral Verbal report aware / not aware Decision rule (implicit)

7. Most frequently used experimental paradigms

8. None of the experimental paradigms used to explore the sensorimotor dissociation allowed a trial-by-trial analysis of the relationship between the motor behavior &the state of the perceptual system (Hits, FA, Misses, CR).

9. PART II RESPONSE TIME & PERCEPTUAL STATE

10. 7° NOT Masked 22.5° Masked (metacontrast) S2 S2 d’ modulation S1 S2:Mask/Primed (p = 1) S2:Mask/Primed (p = 1) criterion modulation S1: Target/Prime (p = .2, .5, .8) T0: Start Clock Hits FA Misses CR 0 20 20 20 100-200 SOA: 50-250 SOA: 50-250 Simple RT Correlation S2-RT Correlation S1-RT 400-1000 T I M E S1: Yes/No? Stimuli & Paradigm (one trial) Waszak & Gorea (2004).

11. Of the 4 perceptual response categories, Hits & Misses are of particular interest: • They tell us about the motor behavior when the Obs. says he senses and does not sense the test stimulus (S1), hence establishing the relationship between perceptual and motor behavior*. • * RT for FA are not reliable indices as they have an unknown temporal origin; • RT for CR are simply used as reference for the relevant RT.

12. variable 650 Masked (metacontrast) t SOA 20 ms 20 ms 600 S2 S1 550 SPACE 500 RT (ms) 450 TIME AG p[S1] = .5 400 350 0 1 2 3 4 5 [S1] Hits Misses CR FA 4 Sjs S1 13 ms SOA variable S2 36 ms 300 trials / d’ / Obs • The motor system appears to react if and only if • the stimulus is present • and • the observer is “aware” of it (i.e. only for Hits). • The difference between RTs for Hits & Misses points against a full sensori-motor dissociation. Waszak & Gorea (2004).

13. 650 600 550 RT (ms) 500 450 400 350 650 0 1 2 3 4 5 600 550 RT (ms) 500 450 400 350 650 0 1 2 3 4 5 600 550 RT (ms) 500 450 400 350 0 1 2 3 4 5 Hits Misses FA CR FW AG SD p[S1] .2 .5 .8 Waszak & Gorea (2004).

14. Rank Correlations between RT and S1-, S2-onsets as a fct. of d’ a. Hits b. Misses 0.7 0.7 SPACE r tR-tS1 & r tR-tS2 r tR-tS1 & r tR-tS2 0.3 0.3 S2 S1 S2 Sj FW S1 0 1 2 3 4 5 0 1 2 3 4 5 -0.1 -0.1 SD Correlation S2-RT Correlation S1-RT AG Speeded RT 0.7 0.7 TIME r tR-tS2 r tR-tS2 0.3 0.3 c. Correct Rejections d. False Alarms 0 1 2 3 4 5 0 1 2 3 4 5 -0.1 -0.1 Waszak & Gorea (2004).

15. …with a slightly modified layman’s view A’. Layman’s view modified Lateral Interactions (implicit) Action with perceptual awareness We’ve thus replaced the standard 2-pathways view… B. The 2 pathways view Decision rule not specified Action with or without perceptual awareness ? dorsal Sensory Input ventral Verbal report aware / not aware Decision rule (implicit)

16. Where does the discrepancy come from? (Aside from potential methodological problems in previous studies) Common denominator: Most of the previous (whether simple or choice RT) studies used 100% contrast targets whose ‘invisibility’ was ensured by strong backward masking. In contrast, our targets (S1) yielded maximum contrasts of about 20%. To allow for higher target contrasts while keeping sensitivity constant, shorter SOA-s (48 ms instead of an average of 162 ms) were used in a second series of experiments; these entailed S1 contrasts around 30%.

17. Masked (metacontrast) SOA t S2 20 ms 20 ms S1 450 FW SPACE AG RT (ms) 400 TIME p[S1] = .5 350 p[S1] = .5 0 1 2 3 4 5 0 1 2 3 4 5 Hits Misses 52 ms CR FA Waszak & Gorea (2004).

18. C. Gorea & Waszak (2004) Fixed Motor Threshold Action with or without perceptual awareness dorsal Sensory Input Action with perceptual awareness ventral Verbal report aware / not aware Lateral Interactions Variable Perceptual Criterion The temptation was strong to conclude (Gorea & Waszak, 2004)…

19. Masked (metacontrast) NOT Masked SOA S2 “mask” 52 ms S1 “prime” S2 S2 S2 S1 S1 t 13 ms 36 ms 320 Perceptual HITS RT appear to depend on Contrast rather than on d’! 300 8% 280 13% 13% RT RT for Misses drop with d’ (or contrast) only for the masked condition! 260 27% 240 220 0.5 1 1.5 2 2.5 d' NOT Masked Masked 17 Obs 300 trials / d’ / Obs Perceptual MISSES 13% 8% NOT Masked 13% Masked NOT Masked 27% Masked 0.5 1 1.5 2 2.5 d' Waszak & Gorea, new experiments.

20. However, the data are more intricate than that…

21. Masked (metacontrast) NOT Masked SOA S2 “mask” 52 ms S1 “prime” S2 S2 S2 S1 S1 t 13 ms 36 ms MISSES (6 Obs) 10 MASKED (6 Obs) NOT MASKED (6 Obs) 0 .14 .21 HITS (6 Obs) .45 .32 1.2 -10 .75 2.1 1.8 RT gain rel. to CR (ms) -20 RTHITS-RTCR RTHITS-RTCR 2.6 2.1 d’ RTMISS-RTCR RTMISS-RTCR 3.2 3.3 2.4 300-900 trials / C / Obs 300-900 trials / C / Obs 2.6 -30 2.8 2.8 3.1 3.2 3.2 d’ 3.4 -40 -50 0 10 20 30 40 0 10 20 30 40 Masked Masked S1 Contrast (%) S1 Contrast (%) Not Masked Not Masked 300-900 trials / C / Obs 300-900 trials / C / Obs Waszak & Gorea, new experiments.

22. Masked (metacontrast) NOT Masked SOA S2 “mask” 52 ms S1 “prime” S2 S2 S2 S1 S1 t 13 ms 36 ms MISSES (6 Obs) 10 NOT MASKED (6 Obs) MASKED (6 Obs) HITS (6 Obs) 0 -10 RT gain rel. to CR (ms) -20 RTHITS-RTCR RTHITS-RTCR RTHITS-RTCR RTMISS-RTCR RTMISS-RTCR RTMISS-RTCR 300-900 trials / C / Obs 300-900 trials / C / Obs 300-900 trials / C / Obs -30 -40 -50 0 1 2 3 4 0 1 2 3 4 Masked Masked d’S1 d’S1 Not Masked Not Masked 300-900 trials / C / Obs 300-900 trials / C / Obs Waszak & Gorea, new experiments.

23. …we propose: D. Current view C. Gorea & Waszak (2004) Fixed Motor Threshold Fixed Motor Threshold Action with or without perceptual awareness Action with or without perceptual awareness dorsal ventral Sensory Input Sensory Input Action with perceptual awareness Action with perceptual awareness ventral ventral Lateral Interactions Verbal report aware / not aware Verbal report aware / not aware Lateral Interactions Variable Perceptual Criterion Variable Perceptual Criterion …And a simpler, one pathway SDT model can account for them. Instead of:

24. A conceptual model

25. Masked (metacontrast) NOT Masked SOA S2 “mask” S1 “prime” S2 S2 S2 S1 S1 t 7 SOA SOA 6 S1 d’ S2 5 S2 4 RESPONSE (Noise units) 3 Perceptual Misses Perceptual Misses Motor threshold 2 S1 S1 1 d’ 0 40 140 240 340 40 140 240 340 -1 TIME (ms) TIME (ms) RTd’=1 RTd’=1 RTMask RTMask -2

26. SOA NOT Masked Masked (metacontrast) S2 “mask” S1 “prime” S2 S2 S2 S1 S1 t SOA SOA d’ Perceptual Misses Perceptual Misses Motor threshold d’ RTd’=2 RTd’=2 RTMask RTMask

27. SOA NOT Masked Masked (metacontrast) S2 “mask” S1 “prime” S2 S2 S2 S1 S1 t 7 SOA SOA d’ 6 5 Perceptual Misses 4 Perceptual Misses 3 RESPONSE (Noise units) Motor threshold d’ 2 1 0 -60 40 140 240 340 -1 TIME (ms) RTd’=4 RTd’=4 RTMask RTMask -2

28. Masked (metacontrast) NOT Masked SOA S2 “mask” 52 ms S1 “prime” S2 S2 S2 S1 S1 t 13 ms 36 ms 300 NOT MASKED 6 Obs MASKED 6 Obs 290 280 270 RT (ms) RTHITS-RTCR 260 RTMISS-RTCR 300-900 trials / C / Obs 250 240 230 0 1 2 3 4 0 1 2 3 4 d’S1 d’S1 Model Fits Motor Threshold .8s

29. A. Layman’s view B. The 2 pathways view Decision rule not specified Action with or without perceptual awareness ? Action with perceptual awareness dorsal Sensory Input Sensory Input Perceptual Decision ventral Verbal report aware / not aware Verbal report (aware / not aware) Lateral Interactions implicit Decision rule implicit C. Gorea & Waszak (2004) D. Current view Fixed Motor Threshold Fixed Motor Threshold Action with or without perceptual awareness Action with or without perceptual awareness dorsal dorsal ventral Action with perceptual awareness Action with perceptual awareness Sensory Input Sensory Input ventral ventral Lateral Interactions Verbal report (aware / not aware) Verbal report (aware / not aware) Lateral Interactions Variable Perceptual Criterion Variable Perceptual Criterion

30. TAKE-HOME MESSAGES (Part II) • A one-pathway model with two distinct activation levels accounts for the observed perceptual-motor relationship under both masking and non-masking experimental conditions. • There is a fixedmotor threshold ( 0.8s)  to be contrasted with a variable perceptual criterion). • The motor threshold is measured in noise (s) units as referred to the “absolute” perceptual detection “threshold” (i.e. in ref. to the internal noise). • RT for “unconscious” stimuli (i.e. Misses) depends on the reference (noise) levelat which the perceptual task is performed; • as this reference level exceeds the motor threshold, the internal response associated with perceptual Misses also exceeds it and progressively contributes to shortening the RT.

31. PART III RESPONSE TIME & TEMPORAL ORDER JUDGMENTS

32. mTrigger-Delay  PSS C1 C2 s Internal Response p(S1 perceived first) Threshold slope PSS PSS SOA mRTC2 mRTC1 Time Temporal Order Judgments & RT IFF RT is strictly dependent on the sensory signal (as it determines the TOJ), then the slope and the PSS of the TOJ -function should be direct indicators of the variance of the RT distributions and of their mean difference, respectively: this is a one-pathway sensorimotor model.

33. DRT DG 500 ms C2 500-1300 ms 1 DG 0.9 SOA C1 1700 ms 0.8 RT 0.7 C1-C1 ~ C2-C2 0.6 C1-C2 %S1 first 0.5 PSS = 0 ms PSS = 49 ms 0.4 0.3 0.2 0.1 Left / Right 0 -200 -100 0 100 200 SOA (ms) Stimuli & Paradigm (one trial) • In contrast with previous studies: • TOJ and RT were measured in the same trial; • It was hence possible to assess RTs for correct & incorrect TOJs. a b c

34. Correct / Incorrect m-RT Incorrect Identical Correct S1 1st Different S2 1st

35. Identical Different Correct / Incorrect Correct / Incorrect m-RT m-RT 375

36. m a b Red: C1 Green: C2 Blue: O1 Brown: O2 s c d

37. DRT a b R R CHigh CLow Mq PC PSS PSS PT1 PT2 RT RT1 RT2 Mq < PC Mq = PC Mq > PC DRT t

38. TAKE-HOME MESSAGES (Part III) • For 2 out of 4 Obs, RT to single stimuli also predict their TOJ behavior (PSS and slopes of the TOJ Y-fct.); their behavior is hence compatible with a one pathway model (no sensorimotor dissociation). • For these 2 Obs, the relationship between RT-differences and PSS also suggests the existence of a motor threshold distinct fromand higher than the perceptual criterion. • The behavior of the remaining 2 Obs does not follow any coherent pattern; it is likely that it was perturbed by perceptual response strategy factors, possibly under the influence of “transient Troxler masking” (Kanai & Kamitani, 2003). • Such factors may account for the various inconsistencies in the RT-TOJ literature.

39. GENERAL CONCLUSIONS • The relationship between simple Response Times and perceptual states (Hits, FA, etc. and Temporal Order Judgments) can be accounted for by a single (one-path) system where two distinct decisions are made on the same incoming information. • The decision to act is based on a hard-wired thresholdworth about 1s of the internal noise; the perceptual criterioniscontext dependent(in line with SDT). • TOJ data not complying with this model may reflect variable perceptual response strategies.

40. THANK YOU

41. SPACE =< A vs. Most likely both (also depending on SOA) => TIME B vs. Either, both? C What do the Obs. compare? OR OR