The Problem and its Significance Premises :

1. Problems of Attention Detection and Prediction for Subjects Interacting with Transportation SystemsProf. Ing. Mirko Novák, DrSc.Czech Technical University, PragueFaculty of Transportation Sciences,Department of Control Engineering and TelematicsLaboratory of System ReliabilityKonviktská 20, Prague 1, 11000e-mail: mirko@fd.cvut.cz

2. Abstract:In this contribution the main problems are discussed, which one has to face when wish to detect, analyze and predict the decrease of human subject attention in the course of his/her interaction (control, use) of artificial (namely transportation) system, having in the mind the need for the development of practical and function reliable on-board applicable operator and/or his/her supervisor warning system.

3. Problems of Attention Detection and Prediction for Subjects Interacting with Transportation Systems Content: 1. The Problem and its Significance 2. Possibilities of Solution 3. Results and Applications 4. Expectance

4. Problems of Attention Detection and Prediction for Subjects Interacting with Transportation Systems • The Problem and its Significance Premises: - Till now, there is no artificial system at disposal, which can operate or be used absolutely without any interaction with human subject. - Human subject can deal with artificial system without faults only fir limited time, after that his/her attention decrease and the danger of accidents caused by wrong interaction increases. - Statistics tell us that e.g. on roads 10 – 15% of all accidents is caused by decrease of drivers attention. In the Czech Republic this leads to losses of about 1 billion CK per year. Problem: Insufficient reliability of system operators interaction causes extremly high losses on health and money – especially as concerns the transportation. The problem consists in finding the way how to minimize or even prevent this.

5. Problems of Attention Detection and Prediction for Subjects Interacting with Transportation Systems The goal: To find methods and tools for detection, analysis and prediction of transport system operators attention decrease (even micro-sleeps) and to develop reliable and practically applicable warning systems.

6. Problems of Attention Detection and Prediction for Subjects Interacting with Transportation Systems Requirements: • Non-invasive for operator • Speed of operation • Operation reliability • Safety • Easy use • User friedly • Satisfactory long life-time • Acceptable investment and operation price

7. Problems of Attention Detection and Prediction for Subjects Interacting with Transportation Systems • Possibilities of Solution: - the use of so called secondary significant markers of attention (eye movement frequency, temperature, skin el. resistance, face character etc.) advantage: easier practical use disadvantage: insufficient specificity, high time delay - the use of primary markers of attention (EEG, EMG): advantage: minimal time-delay, high specificity disadvantage:complicated practical use.

8. Problems of Attention Detection and Prediction for Subjects Interacting with Transportation Systems • Results and applications: After abot 5 years of research (team LSR FTS CTU, ICSCAS, MA Brno, Neurologicalclinic 1.MF UK etc) is: - at disposal and laboratory tested methodology for in-time detection, analysis and prediction of attention decrease of artificial systems operators in the course of their service based on EEG analysis - formulated the necessary background of specific data-base of results of coordinated measurements of EEG signals and probands reaction (Micro Sleep Base – MSB) - developed the concept of the warning system

9. Problems of Attention Detection and Prediction for Subjects Interacting with Transportation Systems Open questions: - the creation of satisfactory wide data-base for generalization of the obtained results, - the development of satisfactory reliable and low power consuming transmission tool between the head detectors and on-board equipment - development of EEG detection set minimizing the discomfort of operator - verification of long time applicability of the specific individual tuned classificators and predictors of attention decrease and micro-sleeps and possibilities of their eventual typization - finding of optimal ways for operator warning with respect to specific operation conditions - solution of specific ethic and legal problems connected with exploation of operators and probands personnal data

10. Problems of Attention Detection and Prediction for Subjects Interacting with Transportation Systems 4. Expectance a) In the national (CNNN – Czech National Node for Neuroinformatics) and international cooperation(Global Science Forum OECD) the Micro Sleep Base – MSB will be filled out by satisfactory amount of relevant data b) The methodology and tools for economic, ethic and legal acceptable use of long time applicable individual oriented prediction and warning against decrease of interaction reliability of operator dealing with artificial system will be developed c) The practical aplicable and economic acceptable on-board warning systems will be developed and set into wide scale use.

11. Introduction No of these many artificial systems, with which human society has daily to deal is able to operate quite independent – all of them up to now have to interact with human subject, i.e. to be controlled, supervised or at least used by man. The quality and reliability of such interaction is of top importance for all of us, without respect to the level of the particular artificial system intelligence. Unfortunately the very long experience of human dealing with artificial systems lead to observation, that very often the human ability is the weakest point in such interaction.

12. The reason is quite natural and easy to understand: • On contrary to artificial systems, human cannot operate too long without brake - he needs to relax, rest and to sleep. The limitation and in the course of his/her operation service or usage of the respective system decreasing human vigilance and attention was and still is the most frequent reason of many system failures and accidents. • Another source of system failures has to be seen in the possibility, that the human operator (or user) of particular artificial system can react late and that his/her decision and choice of kind of reaction must not be correct. • Human behavior is not fully deterministic; it varies from subject to subject.

13. All these factors combine with the result: the reliability of human subject – artificial system interaction is limited, dominantly from the human side.

14. The price, which all we pay for the not satisfactory reliable human – artificial system interaction (non-satisfactory before all from the human side) is tremendous. More over: it increases by time and by level of artificial system sophistication.

15. We urgently need to do something against it. There are various approaches how we can try to minimize the hazard of system operation failures.

16. In general there are the following main ways of improvement the system reliability: • -One of the oldest one consists in the tendency to make the system as much robust, preferring the use of maximal possible reliable (and therefore large, massive and also expensive) components for its construction. • -Another approach is based on the duplication and often even the multiplication of the most important parts of the system under consideration or even of the duplication or multiplication of the whole systems, which eventually can operate as “hot” reserve. Of course, like the first mentioned approach, also this can require very large effort. • -More sophisticated approach tries to modify the system structure so, that the sensitivity of its system functions to the parameter changes will be minimized. - In very sophisticated systems the approach of the predictive diagnostics can be applied, by which one can analyze, how much and in what direction the values of some system parameters can deviate from their nominal values so that the system as a whole is still of acceptable properties and to predict with which probability and when these limits of acceptable system behavior will be broken.

17. Actually, no of these four main approaches is universal, in praxis they are usually combined.

18. As concerns the reliability of interaction among human subjects and artificial systems, only the approach of predictive diagnosticsseems to be prospective for to be candidate of really effective minimizing the frequency of system operation failures caused by human errors and miss-functions, though the first mentioned approach, represented in the area of human subject the tendency of selection the best qualified people from all candidates for operators and in the improvement of their training and testing is also very important.

19. In this talk, devoted to the problems related to the technical aspects of the reliability of interactions between human subject - either in the role of operator (driver, pilot, dispatcher), supervisor or user - and the artificial, namely technical (and before all transportation) system, the final attention is given dominantly to the possibilities how to create a practically applicable tool for diminishing the losses, which are up to now daily caused by decrease and limitation of such interaction reliability.

20. The problem of interaction reliability among artificial systems and human beings (operators, users), though extremely important, is still not solved satisfactorily even the control of dominant part of complex and complicated systems is now realized by computers or at least by computer assistance.

21. The human operator, who has to interact with powerful, complicated and often also efficient artificial system (such as transport system, aircrafts, express trains, large trucks, extensive power stations, security and defense systems etc.) is imposed to requirements on fast and correct reactions on very variable actual situations and he/she is exposed to this for considerably long time of his/her service. The resulting high load of such operator brain and nervous system results necessarily to decrease of his/her vigilance and to degradation of his/her attention.

22. The complex demands on human operator interacting with complicated and powerful artificial system are more over often jointed with influence of some internal and external factors (such as long length of service, psychical isolation in the course of service, operators actual mental and physical state, climatic conditions, quality of environment in the particular cock-pit or control room, monotony of the scene or image, which operator has to observe etc).

23. Also some psychical load of the respective system operator and the presence of pre-service or in the course of the service acting stress factors can play very important role.

24. In this respect we have to take into account also various other circumstances, typical for humans, namely: - The extremely high variability of human subjects, especially as concerns their intelligence, education, level of operation training and skill, ability to concentrate, tendency to no rational and panic reactions on unexpected and suddenly appearing situations, general temper, tendency to be aggressive etc. - The human subjects dealing with artificial systems alsodiffer very much in age, race and sex. - In contrary to artificial systems, the reaction of human subjects cannot be exactly repeated – the human subject learns from experience and modifies his/her behavior to minimal expense of physical and psychical energy, which he/she has to use for system control or use.

25. All these factor complicate seriously the problem – we cannot use the standard methods of control and system engineering. Nevertheless we need to transfer the methodology from standard psychologically oriented approaches used for decades in system operator education and training to more contemporary ones, dealing with much more knowledge of the field of system reliability theory, mathematical analysis, neurology, artificial neural networks, numerical methods etc.

26. 2. The role of interaction with human subject in artificial system operation Artificial system can interact with human being in following principal ways: -      - Interaction on the basis of human control of system operation, -      -Interaction on the base of human supervision of system operation, -      -Interaction on the base of human use of system operation, -    -Interaction of human society on system operation.

27. It is evident, that in all these cases the interaction failurescan cause fatal situations, or at least serious economic losses. However, also in the case, when the interaction among some artificial system and various members of human society forming its environment is considered, the interaction reliability and predictability could be of very high importance. This concerns especially such situations, when the artificial system changes suddenly its behavior and when it interacts with considerably large and heterogeneous part of human society. The acceptable reliable estimation of most probable kind of such environmental reaction needs deep understanding not only of individual behavior of human subject exposed to interaction with varying properties of particular artificial system, but also of the possible social factors, which exist or which could be activated in the respective part of human society. Such studies are evidently of top importance for general safety, however are very difficult and laborious and require the access to satisfactory large special databases in which the results of many measurements of human subject interaction reliability markers are stored. The later on mentioned Micro Sleep Base represents an example of one of them.

28. Recently, the technological progress of the reliability of artificial systems has greatly improved. Consequently the probability of technical faults in well-designed and well -manufactured artificial systems is now usually very limited (though unfortunately still not excluded). However, the probability of failure caused by misuse and faults in system operator activities increases rapidly.

29. The main reason for this not acceptable situation can be seen: • in increasing complexity of the operated and used systems, • in increasing requirements on the operator’s ability to deal with them, • in the increasing requirements on his/her level of continuous and long- time attention and • in the increasing requirements on the speed of his/her reactions.

30. Naturally, the losses caused by artificial system operation faults are proportional to their power, significance and value. In the case of many modern transportation systems (large planes, fast trains, large ships, trucks), large power plants, important financial systems, security and defense systems, and also important medical care systems, the losses caused by their malfunction could be extreme high or also of catastrophe character.

31. Therefore, besides the continuing interest in diminishing the probability of technical failures in any artificial system as much as possible (with respect to economically acceptable expenses), considerable interest has also been shown in recent years in the reliability of system operator activity. Many statistics demonstrate that the amount of human error represents a still larger proportion of all the expenses, which are required for the compensation of artificial system malfunctions.

32. The requirements on a human operator of an artificial system can be concentrated in the following main categories: a) Requirements on attention level and continuity, b) Requirements on the speed of operator reaction, c) Requirements on the correctness of operator decisions.

33. Suppose that all three above mentioned kinds of requirements can be expressed as the limits, outside which cannot be accepted the real number xi of some significant parameter xi, i = 1,2,3 by which we express the level of attention, the speed of reaction and the correctness of operator decision. The acceptable values of the vector X = {x1, x2, x3} fill out in the space X some region RA, called the region of acceptability. The concept of regions of acceptability is well known in the theory of tolerances of system parameters. In the theoretical case, that the parameters xi are independent, the region RA has rectangular form. However, in practice, among x1, x2 and x3 certain correlations exist. This leads to much complicated form of RA.

34. Let us suppose, that we deal with so called well operated systems only, i.e. with such of operator - system interactions, which work well at least of the beginning of their use or of their observation.

35. One can formulate therefore the following theorem: Each real interaction between human operator and artificial system, when exposed to the influence of a set of independent variables P (before all of the time P = pt), can be represented by some position of the vector X in the parameter space, moving along certain interaction life-curve . For t , the life-curvebreaks at least once the boundaries of RA.

36. Fig.1-The idealized run of the interaction life-curve of operator – system interaction inside and outside the respective region of acceptability.

37. In accordance with the general theory of system reliability, we can define The reliability of some operator - system interaction is represented as as the probability H that for certain time interval (or some interval of other independent variable influencing the system under consideration) the respective interaction life-curve (t) will remain inside the region of acceptability RA .

38. Similarly: Safety of the operator - system interaction is to be considered as the probability that it will be resistant against the eventual disturbing influences.

39. A straightforward correlation exists especially between attention level and speed of reaction. Operators commanding at high level of attention usually also possess very fast reactions. On the other hand cases can appear, when fast, almost impulsive reaction may not be accompanied by very high level of the operator’s concentration and attention. Some people can react fast also, when their attention is dispersed on very different objects (they have very fast but unreliable reflexes).

40. A drop in the attention level of a particular human operator can be caused by various external or internal reasons; some of them have a general character; the intensity of others depends significantly on the operator’s individuality. Among the general conditions causing the decrease in attention is: • - Extreme length of a particular operator’s service without breaks, • - Operator’s physical and mental exhaustion, • - Monotonous scene, which the operator has to observe for a long time, • - Extreme temperature in which the operator has to serve (too high or too low), • - Extreme humidity in which the operator has to serve (too high or too low), • - Extreme air pressure, • - Air smell, dust density etc.

41. In practice, the shape of RA and life-curve ψ(t) is much more complicated, as it is shown in the schematic Fig. 1. Before all, the operator attention can improve in the course of his/her interaction with the system even if he/she fall already in the stage of real micro-sleep. In such case the respective interaction life-curve ψ(t)returns inside RA and after some time the procedure of attention decrease starts again. Such episodes, schematically expressed in Fig. 2 for two repetitions, can repeat several-time, until successful end of particular operator service in fortunate case, or until accident in negative case.

42. Fig. 2: The repetition of braking RA boundaries by (t) in two temporary and one final micro-sleep episodes.

43. The number of possible repetitions of such braking RA boundaries in micro-sleep episodes depends on many factors, including the individuality of particular operator, his actual mental and physical state, the environmental and technical conditions of system operation and control etc. Though this represents a very important problem, the contemporary knowledge in this respect is far from to be called satisfactory and much more research in this area needs to be done.

44. Like in many cases when the regions of acceptability of technical systems operation are investigated, also here very often the boundaries of the regions of acceptable interaction can be of fuzzy character.

45. 3. Human vigilance and attention and ways to their decrease classification and prediction Micro-sleep is usually characterized as such a state of an organism, in which the eyes are closed and vigilance approaches zero. On the contrary, one can understand by micro-sleep also such a state of the organism, when its vigilance decreases below a certain limit, without respect to the activity of visual tract. There are also several other conceptions of micro-sleep between these two limits.

46. Evidently, for the purpose of dealing with man-system interaction reliability, the aspect of decrease of vigilance is much more significant than closing of eyes, but the break of visual information input longer than certain limit (about 1 second for car drivers e.g.) can be also dangerous. The following statement has been considered by the opportunity as a reasonable basis for the conception of micro-sleep, as it will be used in further discussion: Let suppose that the level of a human operator’s attention can be measured by some real figure of merit LAT. As it is schematically shown in Fig. 3, the level of attention LAT decreases with the time, in which the human subject mental activity in the course of his/her interaction with the artificial system is observed.

47. Here we can distinguish the following four basic stages: a) Stage of the vigilance (full attention), in which the respective subject is completely competent to control (or use) the system under consideration, b) Stage of relaxation, in which the respective subject is still competent to deal with the system under consideration, however where his/her attention decreases subsequently. This stage can last for considerably long time. c) Stage of somnolence, in which the competence of the respective subject to interact with the system under consideration becomes to be restricted. This stage can last also considerably long, however in contrary to the previous one, here the real danger of the respective subject control faults exist. d) Stage of hypnagogium, in which the respective subject fall into micro-sleep, at first with open eyes, however with very limited ability to control the system under consideration, later on into the micro-sleep with closed eyes, in which the respective subject competence to control the system is almost zero (some very skilled drivers - namely the professional truck drivers - can also in this stage hold the vehicle in straight move, however they cannot adequate react to any road curvature or barrier on the road).

48. Fig. 3: The decrease of attention in the course of human subject activity

49. In general, one can define the micro-sleep as follows: Micro-sleep is such a state of the human organism, in which the mental vigilance and attention of the human operator dealing with some artificial system (operating,controlling or using it) decreases for below a certain limit.

50. Micro-sleeps episodes can be of various lengths. For each kind of human subject interaction with artificial system some limits of maximum acceptable decrease of human subject attention and of micro-sleep lengths exist, over which the particular micro-sleep must be considered as dangerous.