Considering cognitive aspects in designing cyber-physical systems :

1. Considering cognitive aspects in designing cyber-physical systems: an emerging need for transdisciplinarity Wilfred van der Vegte and Regine Vroom Delft University of Technology Faculty of Industrial Design Engineering Department of Design Engineering

2. Faculty of Industrial Design Engineering

3. Contents • Cyber-physical systems • CPSs design – involved disciplines • Disciplinary approaches: mono, multi, inter, intra, and trans • Flavours of transdisciplinarity • Two directions of research: • Simulating cognitive loads and processing times • Informing systems and mental models • Concluding remarks

4. Cyber-Physical systems (CPSs) CPSs are integrations of computation with physical processes, in which embedded computers and networks monitor and control the physical processes, usually with feedback loops where physical processes affect computations and vice versa. Example of a CPS:Swarming Micro Air VehicleNetwork (SMAVNET) @ EPFL, CH • Rapidly creates communicationnetworks for rescuers in disasterareas • Sensor networking technologies • Swarm intelligence

5. CPSs design @ IDE– involved disciplines • Industrial Design Engineering (IDE), • Cognitive Psychology, • Psychophysiology, • Information and Communication Technology (ICT) • Disciplines commonly involved in an interdisciplinary faculty of IDE such as: • Materials technology • Manufacturing technology • Human factors • Electronics • Mechanical engineering • Marketing • etc.

6. Addressing cognitive aspects • In predecessors of CPSs (mechatronic/smart systems, etc.) ICT and physics were already heavily involved.CPSs will increasingly incorporate (distributed) artificial cognitionin interaction with human cognitionHandling cognitive psychology issues will be a key challenge in the near future of CPS development • Cognition-related issues: • Allocation of cognitive tasks between human and CPS • Cognitive matching of inputs/outputs between human and CPS • Preventing information overload of human users • Enabling CPSs as safety-critical systems • Objective: cognitive symbiosis between human and CPS

7. Transdisciplinary vs. intra-, inter- and multidisciplinary Flavours of transdisciplinarity inter other area of science (e.g., medical) engineering science multi mono trans intra intra inter other area of development (e.g., healthcare) engineering design multi mono mono intra intra end users/consumers (e.g., product users) end users/consumers (e.g., patients)

8. Flavours of transdisciplinarity other area of science (e.g., medical) engineering science transdisciplinary design = transdisciplinary research other area of development (e.g., healthcare) engineering design end users/consumers (e.g., product users) end users/consumers (e.g., patients)

9. Two directions of research • Simulating cognitive loads and processing times • Informing systems and mental models

10. 1.0 Simulating cognitive loads and processing times • Key application area: deployment of CPSs as safety-critical systems • Revision of decision-making responsibilitiesCPS  human • Simulation of human mental processes together with models of products and systems (in particular, CPS) • Goal: evaluate CPS during development identify bottlenecks to be addressed in the CPS’s designincluding service design, task design/allocation

11. 1.1 Human-cyber-physical systems – how can we simulate? • Interactive simulation vs. fully virtual simulation: • Safety-critical systems  identification of incidents happening once in ~1,000 years. • Interactive human-in-the-loop simulation must be real-time,but we cannot run a simulation for 1,000 years! •  we need faster-than-real-time simulation  fully virtual, even humans human CPS; environment Use simulation tools common in embedded systems engineering (procedural logic, state machines) How to simulate human thinking and human reasoning? information processing Avoid time-consuming physics simulations based on geometric discretisation (e.g., FEM): use simplified models instead. physics Take shortcuts: disregard perception, motor skills, etc.

12. 1.2 Simulating human thinking and human reasoning • Two aspects:logic of decision making and processing time of decision making • Logic of decision making: • What action is taken under what condition?e.g. “IF cup is full THEN retrieve cup from machine”:straightforward execution of ‘normal’ use,assuming a particular history of preceding events. • But can a simulation predict a user acting according tothe production rule “IF cup is full THEN stick finger in it”? →unlikely! • Yet we can try to generate typical aberrations from ‘regular use’:so-called error phenotypes (Hollnagel):actions accidentally in wrong order, accidental repetition, etc.,by applying systematic variations

13. 1.3 Simulating human thinking and human reasoning: processing time • Processing time:How long does it take to accomplish a given action, taking into account aspects such as memory retrieval, memory capacity, learning, multitasking, distraction, etc. • These aspects can be simulated using cognitive architectures such as ACT-R • A cognitive architecture is • a blueprint of the human mind • based on findings from brain science • filled with psychologically validated task modelsexpressed as production rules

14. 1.4 ACT-R cognitive architecture ACT-R simulation (human) intentional module declarative module ( temporal central production ( not identified ) cortex / hippocampus ) • ACT-R models are task specific, programmed in LISP by skilled, dedicated cognitive scientists • Most tasks require scientists to create new customized models, that have to be validated in laboratory experiments with human subjects • Intensive collaboration between cognitive scientists and designers of CPSs seems inevitable system goal buffer ( dorsolateral retrieval buffer ( ventrolateral ( basal ganglia ) prefrontal cortex ) prefrontal cortex ) motor module ( motor motor buffer visual buffer visual module cortex / cerebellum ) ( motor cortex ) ( parietal cortex ) ( occipital cortex ) simulation ofCPS & environment external world

15. 1.5 Example CPS for simulating cognitive loads & processing times: Advanced support of emergency response

16. 2.0 Informing systems & mental models Informing CPSs (e.g. informing public traffic systems) • aims to find novel means to inform users and to find new symbiotic relations between human and cyber-physical systems; • based on which designers can be supported in the early stages of CPS development; • the objective is to avoid situations where users are mentally or perceptually overloaded and to precisely give the information that will help to take a right decision to react.

17. 2.1 Project purpose The purpose of this project is to gain a better understanding in the manner in which MMs influence our interaction with the informing part of CPSs, and to provide guidelines for designers based on these insights Current detection e.g. through motion detection, smart phone connection, id tag, … Human output detected by a CPS Cyber-Physical System Human “system” Human output Current situation Brain (cognition, including knowledge, experiences, reasoning) CPS input Processing Sensors /detectors CPS outputadapted to the cognitive capabilities of individual user(s) in a specific situation Human input CPS informs (or offers other functionality) to human Senses

18. 2.2 Informing systems & mental models Goal: Include cognitive insights to influence the adaptability of CPSs. Approach: Study the behavior of mental models Cyber-Physical System Human “system” Mental model: internal representation that people hold of an external reality that allows them to explain, interact, and predict that reality (from cognitive psychology) Human output Future situation Brain (cognition, including knowledge, experiences, reasoning) CPS input Processing Mental model Sensors /detectors Mental model CPS outputadapted to the cognitive capabilities of individual user(s) in a specific situation Human input Senses

19. 2.3 Future situation Cyber-Physical System Human “system” Human output Current situation Designerly cognitive insights Brain (cognition, including knowledge, experiences, reasoning) CPS input Processing Mental model Sensors /detectors CPS outputadapted to the cognitive capabilities of individual user(s) in a specific situation Human input Senses precisely give the information that will help to take a right decision to react Design Engineering CognitiveScience

20. 2.4 Designerly cognitive insights • Study behaviour of mental models: • Is there inertia when switching from one mental model towards another? • E.g. if an unexpected situation occur, will there be a different reaction on the same situation if the person was reading an exciting book than when he was playing football? Difficulty: perception influences can’t be reset (“undo” or “delete”) • How to identify inaccuracies and gaps in a mental model (i.e. in a person’s knowledge and experience)? • Mental models are inaccurate and incomplete. Insights in how to determine the gaps and the faults incorporate clues to better inform people.

21. 2.4 Designerly cognitive insights cont’d • Study relationships between mental states and cognition at one side and physical human data (facial expressions, gestures, heart beat etc., i.e. psychophysiology) on the other side. • In addition to search for direct determination methods, indirect measurements might be useful: some facial expression may indicate that a person doesn’t understand a message for example. • How to effectively address the major gaps and faults in a mental model? • Effect of senses to address, effect of amplitude of the message (audio volume, pressure level in haptic information, etc.) The bridge towards guidelines for designers of informing CPSs: insight in the mental model states and behaviour will enable designers to design CPSs with adaptive capabilities on the user’s cognition.

22. Concluding remarks – cognitive modelling • Two research directions • one to allow virtual testing of CPSs by designers, taking into account speed and capacity of human cognition in the interaction with CPSs • one to provide knowledge to designers, so that CPSs can adapt themselves to mental models maintained by their users • Both entail transdisciplinary collaboration with cognitive scientists –from disjunct research communities with different scientific approaches • Cognitive architectures are based on empirical laboratory experiments • Mental models are captured based on interviews • Common goal: achieve symbiotic relationship between human and CPS

23. Concluding remarks – Transdisciplinary design (or: research) Defining what is transdisciplinary and what is not, is probably not as simple as we have suggested: • Knowledge value chain often more complex thanresearch  development/design  application • What is one discipline? How to deal with hierarchies of disciplines?(e.g. engineering electrical, mechanical, civil, ...) • If we have learned enough from working with expert scientists from another discipline, can we eventually do the trick ourselves?Does it mean that a new discipline has formed and the activity is no longer ‘transdisciplinary’? If so, is that bad?