The usability of cardiovascular and electrodermal measures for adaptive automation

1. The usability of cardiovascular and electrodermal measures for adaptive automation A. Haarmann, F. Schaefer, W. BoucseinPhysiological Psychology University of Wuppertal, Germany (Director: Prof. Dr. W. Boucsein)

2. Monitoring automated processes vigilance task with low requirements reduced workload resulting from low demands vigilance decrement Manual control in situations the system was not programmed for responding to sudden errors within a small time window heightened workload resulting from excessive demands Background: Shift in requirements of system operation  “Ironies of automation“ (Bainbridge, 1983)

3. Critical questions in highly automated systems (Wiener, 1989) • What is the system doing? • Why is it doing that? • What will it do next? • Is the operator in or out of the loop (Endsley, 1996)??? • “automation surprises” (Sarter, Woods & Billings, 1997) • “going sour” accidents (Cook, Woods & McDonald, 1991)

4. Adaptive Automation • Capability of a system to adjust its mode or increasing/reducing the degree of automation dynamically dependent on psychophysiological arousal measures: • Hypovigilance/underarousalthe system may alert the operator, thus increasing his/her attention • Operator’s workload too high/overarousal the system may automatically take over more responsibility for the task in question

5. Prerequisites for Adaptive Automation • Continous monitoring of psychophysiological measures • On-line evaluation of these measures • No interference with the operator‘s well being • Closed loop control

6. Open- vs. closed-loop control(Sheridan & Ferrell, 1974) Closed-loop control: The difference between ideal and actual system response is measured and used as an imput to an otherwise open-loop device to drive the system in the direction of - reducing the error - counteracting disturbances.  servomechanism(e.g. a thermostat-controlled heating system)

7. Adaptive Automation • First studies on adaptive automation: • Adaptive closed loop based on EEG frequency bands derived by power spectral analysis (NASA research group with Parasuraman and Prinzel ) • Recently: heart rate variability (e.g. Prinzel et al., 2003) • electrodermal activity (Yamamoto & Ishiki, 1992)

8. Adaptive closed loop: An example based on NS.SCRs* 1 Controlling a flight simulator (5<8 NS.SCRs) monotony, habituation, orientation set point individual baseline of mean arousal, e.g. 8 NS.SCRs/min deviation 3 2 disturbance factors criterion for regulation actual value 4 subject faces higher demands in order to rise his/her NS.SCRs (e.g. with turbulences) Recording and on-line-parameterization of EDA, ECG, RES, e.g. 5 NS.SCRs/min 5 next recording period * NS.SCRs = nonspecific skin conductance responses per time unit (e.g. 1 min) as a tonic electrodermal measure for arousal

9. Own studies on adaptive automation • Pilot study (Boucsein, Haarmann & Schaefer, 2005): • Aim: Checking usability of peripherical physiological measures as potential candidates for an adaptive closed loop • Flight simulator LAS 5.0 (Fahsig, Germany) • 16 student subjects flew four missions four times, each time under a different stage of turbulences. • Sequence of turbulences counterbalanced according to a Latin Square • Recording of electrodermal activity (EDA), heart rate (ECG) and respiration

10. Cockpit of LAS 5.0 (Fahsig) magnetic compass horizontal situation indicator (HSI) NAV indicator (NAV1) airspeed altitude COM and NAV settings directional gyro turn coordinator vertical speed rev meter (rpm) tanks tank selector

11. Subject “at work“

12. Electrodermal activity Electrocardiogram Respiratory activity EDA electrodes: Ag/AgCl, 8 mm diameter, thenar/hypothenar, left, isotonic electrode cream Electrodermal recording: with 20 Hz sampling rate and .001 µS sensitivity, using .3 Hz low pass filter Electrodermal activity (SC): frequency and sum amplitude of nonspecific SCRs, amplitude criterion .01 µS

13. F (3, 36) = 3.05, p = .06 NS.SCRs during different stages of turbulence (frequency for 2-min-intervals) increment  increase of workload

14. F (3, 36) = 4.21, p = .03 NS.SCRs during different flight missions (frequency for 2-min-intervals) decrement  habituation (familiarization with flight task)

15. F (3, 36) = 10.49, p < .001 Heart rate variability during different stages of turbulence (bpm during 2-min-intervals) decrement increase of workload

16. F (3, 36) = 11.16, p < .001 Heart rate variability during different flight missions (bpm during 2-min-intervals) decrement  fatigue

17. Own studies on adaptive automation • Based on previous pilot study (Boucsein et al.,2005):Implementation of adaptive automation using electrodermal activity during a simulated IFR flight mission: • Aim: Adaptive adjustment of physiological arousal by online-recording and parameterization of tonic electrodermal activity • Default value for adaptive automation: Frequency of nonspecific skin conductance responses (NS.SCRs)

18. 18 student subjects, aged 20 to 34 years (M= 26.39 years, SD= 4.5 years) 11 females  7 males as part of a psychology course requirement Sample

19. Flight task • Familiarization with flight simulator LAS 5.0 (Fahsig, Germany). • Flight missions including: • Take off and climbing to 2000 ft • Flying straight and level northbound, controlling altitude, speed and course • Turning 90 degrees eastward and keeping that course, controlling altitude and speed • Keeping new course, controlling altitude and speed while facing turbulences (turbulence steps 0 and 2).

20. LAS computer: Presentation of the flight simulator soft-ware LAS 5.0with external control of weather impacts(e. g. turbulences) Control computer:  triggering the automatic on- and off-set of turbulences on the LAS computer starting physiological data recording on a third computer receiving the on-line calculated NS.SCRs from the recording computer for adaptive regulation of the subject’s arousal SUBJECT („pilot“) Recording computer with polygraph and AD converter:Recording of psycho-physiological parameters followed by on-line parameterization of EDA indicator of arousal (NS.SCRs)

21. Procedure • Two baseline recordings for set point calculation • 60 s without turbulences as resting period • 60 s with turbulence step 2 as workload period • arithmetic mean of baseline recordings as individual set point • 30 flight sections, updated every 60 s according to the adaptive algorithm: • actual value < set point  subject underaroused  onset of turbulences (step 2) • actual value > set point  subject overaroused  offset of turbulences (step 0)

22. Procedure • Two conditions: • experimental („adaptive“) condition on- and off-set of turbulences dependent on the subject‘s individual set point • yoked-control condition  subject received the same sequence of turbulences as the corresponding experimental subject, regardless of his/her own set point

23. Subject 108 (adaptive condition)set point = 9 NS.SCRs Mean deviations: t(29) = 2.394, p < .05 Subject 208 (yoked control)set point = 8.5 NS.SCRs flight sections (60 s per section)

24. Results • Significantly smaller deviations from the set point in the adaptive condition compared to the yoked control condition • Highly significant correlations of NS.SCRs and resulting turbulences in the experimental group (up to r = -.875)CONCLUSION: • optimal vigilance/workload level as a result of adaptive control in contrast to the yoked control subjects

25. Follow-up study • Comparison of different combinations of autonomic measures regarding quality of regulation • Four instead of two baseline recordings for more accurate calculation of the subject‘s individual set point • Extension of recording periods to 2 min per flight section to account for • slow changes in physiological measures • problem of temporal effects occurring during long-term task performance, e.g. learning, adaptation and fatigue. • Extension of steps of turbulence (not only 0 and 2) for a more fine-grained regulation

26. Extension of turbulence stages • 36 student subjects (18 males, 18 females) • Presentation of the flight simulator‘s six turbulence stages (0 to 5), each step presented for 2 min • Sequence of turbulences counterbalanced according to Latin Square • After each presentation: Subjective estimation of arousal on the „General Central Activation Scale“ (Bartenwerfer, 1969) • vertical scale (0 to 50 points) for comparison of arousal with daily life as well as exceptional situations

27. F(5, 170) = 22.42, p < .001 • ANOVA with repeated measures and post hoc comparisons: Turbulence steps 0, 1, 3 and 5 most suited for the follow-up study.

28. Thank you for your attention!

29. Yerkes-Dodson-Law

30. NASA-Study: arousal feedback using EEGPope et al. (1995) Multi Attribute Task Battery (MAT) *Engagement-Index (EI) = EI falls below set pointmanual mode EI exceeds set point automatic mode automatic vs. manual MAT Analog Signal Out EEG engagement index* calculated, feedback and task allocation decisions made EEG is collected for later PSA and ERP analyses (Cz, T5, P3, Pz, P4, O1, O2)