Professor Amir Hussain, Dr Andrew Abel 1 Division of Computing Science and Mathematics

1. Cognitive Computation: A Case Study in Cognitive Control of Autonomous Systems and Some Future Directions Professor Amir Hussain, Dr Andrew Abel 1 Division of Computing Science and Mathematics University of Stirling, Scotland Work reported here is part of an ongoing UK EPSRC funded project, with: Dr Erfu Yang1 (RF) & Prof Kevin Gurney2 (CI) 2Adaptive Behaviours Research Group (ABRG) Department of Psychology University of Sheffield, UK The International Joint Conference on Neural Networks (IJCNN) Dallas, Texas, August 4-9, 2013

2. Introduction • Why Cognitive Computation? • Why Cognitive Machines? • Taylor’s Proposal on Cognitive Machines • Cognitively Inspired Control of Autonomous Systems • Towards a more generalised cognitive framework Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

3. Introduction • Cognitive computation • an emerging discipline linking together neurobiology, cognitive psychology and artificial intelligence; • Springer’s journal Cognitive Computation publishing biologically inspired theoretical, computational, experimental and integrative accounts of all aspects of natural and artificial cognitive systems. • Professor John Taylor • founding Advisory Board Chair of Cognitive Computation; • proposed on how to create a cognitive machine equipped with multi-modal cognitive capabilities. • This keynote • first presents a novel modular cognitive control framework for autonomous systems - potentially realizes the required cognitive action-selection and learning capabilities in Professor Taylor's envisaged cognitive machine. • Possible future avenues for improving this work in a cognitively inspired manner Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

4. Why Cognitive Computation? • Promote a more comprehensive and unified understanding of diverse topics • perception, action, and attention; • learning and memory; • decision making and reasoning; • language processing and communication; • problem solving and consciousness aspects of cognition. • Industry, commerce, robotics and many other areas are increasingly calling for the creation of cognitive machines, with ‘cognitive’ powers similar to those of ourselves: • are able to ‘think’ for themselves; • reach decisions on actions in a variety of ways; • are flexible, adaptive and able to learn from both their own previous experience and that of others around them Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

5. Why Cognitive Machines? • A multi-disciplinary research challenge • Understanding our own cognitive powers: • how they are created and fostered; • how they can go wrong due to brain malfunction; • the modelling of the cognitive brain is an important step in developing such understanding. • Creating autonomous robots and vehicles able to ‘think’ and ‘act’ cognitively and ethically: • support us in our daily lives. Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

6. Taylor’s Proposal on Cognitive Machines • It was published at J.G. Taylor, “Cognitive computation,” Cogn. Comput, vol.1, pp.4–16 (2009). • Based on ideas published in many places • Taylor raised a number of very interesting points in his attempts to construct an artificial being empowered with its own cognitive powers: • a range of key questions relevant to the creation of such a machine; • made detailed and methodical attempts to answer these questions; • providing convincing evidence from national and international research projects he had led over the years. • Taylor’s proposal is one of very few attempts to construct a global brain theory of cognition and consciousness. • It is based on a unique multi-modal approach that takes into consideration vision and attention, motor action, language and emotion. • Conventional studies in cognition and consciousness have mostly focussed on single modalities such as vision. Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

7. Taylor’s Proposal on Cognitive Machines • Taylor asked a number of questions • What is human cognition in general, and how can it be modelled? • What are the powers of animal cognition, and how can they be modelled? • How important is language in achieving a cognitive machine, and how might it be developed in such a machine? • What are the benchmark problems that should be able to be solved by a cognitive machine? • Does a cognitive machine have to be built in hardware? • How can hybridisation help in developing truly cognitive machines • Is consciousness crucial? • How are the internal mental states of others to be discerned? • Discussed notion of attention control Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

8. Taylor’s Proposal on Cognitive Machines • This approach to attention control relevant to our interests • Will link to a case study that uses this as a basis for a new approach to autonomous vehicle control • Initially focus on control and decision making • Ongoing work! Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

9. Cognitive Control of Autonomous Systems A Case Study

10. Two problem domains Planetary rovers (SciSys) Smart cars (Google)

11. Challenges in each domain • Urban driving in smart cars • constantly changing trajectories • moderated speed in urban areas • ‘sentinel’ awareness of high pedestrian density • Planetary rovers • real-time trajectory planning for feasible path to follow on • Autonomous navigation • Intelligent motion control with most optimal controller • Active and smart obstacle avoidance • ‘cognitive’ awareness of complex environments Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

12. The problem we tackle:from partially specified trajectories to cognitive control X(t0) X(t1) Construct P(t) subject to smoothness and time constraints X(t2) Path following with error correction Take account of obstacles and challenges

13. The problem we tackle:from partially specified trajectories to cognitive control Vehicle with given dynamics and kinematics Drives along P(t) X(t0) X(t1) Construct P(t) subject to smoothness and time constraints X(t2)

14. Multiple controller methods • Historically • Hard switching • One controller selected at any one time • Issue is ‘bumpiness’ when switching between controllers • Our goal • ‘Bumpless’ control • Soft switching • Select a subset of all controllers • Mix controller decisions together • Smoother output Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

15. Existing hard switching control supervisor disturbance/ noise switching signal s measured output reference input w controller 1 s e(t) r(t) y bank of candidate controllers Plant Model + u _ controller n control signal • Key ideas: • Build a bank of alternative controllers • Switch among them online based on switching condition

16. Compare with the problem of action selection in animals Fight, flight or feeding, but not “do nothing” • The animal solution is centred on a set of sub-cortical brain nuclei – the basal ganglia, which act as a central ‘switch’ or selector • Can we leverage the biological solutions for use in AVC? Basal ganglia in brain Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

17. Ctx1: action1 Ctx2: action2 Thalamus Thalamus BG BG The biology: Disinhibition gating and action channels(compare with modular control) Predisposing conditions Motor resources Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

18. Modular control : Challenges • Meeting multiple performance criteria • Stability • Convergence • Tackling problems of ‘chattering’ • Anti-windup and ‘Bumpless’ switching • Real-time operation Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

19. Three-stage modular framework: a bio-inspired approach Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

20. Using the biomimetic BG model in a control environment 4-wheel rover – Kinematics-based motion control and planning

21. Three-stage modular framework: case study “Actual trajectory” Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

22. Kinematics-based motion control and planning • The motion control of autonomous vehicles is mostly based on the vehicle’s kinematics model • Usually assumed that the vehicle’s internal dynamics can immediately satisfy the velocity/steering angle requests from the kinematics-based motion control • This study: • BG-based kinematic motion controllers are used for motion planning and control • Perfect dynamics assumed Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

23. Kinematics-based motion control and planning Feedback linearisation “actual” trajectory Kinematics to path Two trajectory Components (input from motion planner) Controllers are all Pole placement-based Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

24. Action surface for fuzzy salience model Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

25. Simulation Results • Circular Trajectory Tracking Control (b) x − y trajectory comparison for BG-based switching and a single feedback linearization motion controller under noise (a) States in the circular tracking with BG-based switching and a single feedback linearization motion controller under noises Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

26. C. General Path Tracking – double lane change and roundabout x-y trajectory under BG-based switching and a single feedback linearization motion controller under noises Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

27. Using the biomimetic BG model in a control environment 4-wheel rover – B-Spline path planning and three-stage motion control with integrated kinematics and dynamics

28. Smooth path planning with B-splines • The dimension of the knot vector: 24; • The number of control points: 18; • The degree of splines: 5 Control points and smooth path planned with B-spline method Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

29. General Path Tracking – double lane change and roundabout Comparison of BG-based soft switching control and single-fixed controller with noises Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

30. Comparison of Control Performance (MSE: Mean Squared Error) Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

31. Summary • BG-based controller selection is bumpless‘soft-switching’ because it combines outputs of multiple controllers • We have some evidence that this also helps avoid windup & chattering • BG will allow adaptive control by varying internal parameters which are now better understood from our neurobiological models • Based on model of biological decision making • Attention switching using salience • Ongoing work Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

32. Autonomous Control Specific Future Work • Test against traditional switched controller designs with same controllers • Adaptive online operations • learn salience weights to BG controller • Dynamic allocation of controllers • Use of more realistic models • Real experimental test beds Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

33. Cognitive Future Work • Incorporating vision • Better able to react to world • Use of multiple modalities • Dual process control…. • Automatic behaviour mode • Process known differently from unknown • Learning over time, becomes automatic • Mimics processing in the brain Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

34. Cognitive Computation… …towards a multimodal framework Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

35. More Cognitive Computation? • This is a specific case study • Inspired by work of John Taylor • Cognitive Computation is very wide ranging field of research • Can be applied in many different contexts • Means different things to different people • Presentation tomorrow • Discuss cognitive computation in more depth • Application in more fields • Want to consider a more general cognitive framework Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

36. Sentic Computing • Sentiment Analysis • Common sense computing • Read emotion and tone from text • Traditional approaches inadequate • Machine Learning • Keyword counting • May identify topic, but not sentiment • Concept based approach • Can assign emotions to concepts • Relate similar concepts together Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

37. AffectNet Graph Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

38. AffectiveSpace E. Cambria and A. Hussain. Sentic Computing: Techniques, Tools, and Applications. Dordrecht, Netherlands: Springer, ISBN: 978-94-007-5069-2 (2012) Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

39. Sentic Computing • Sentic Activation • Consider conscious and unconscious level processing • The two interact • Can be used for sentiment analysis • Emotion detection Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

40. Multimodal Speech Processing • Traditional hearing aids focus on single modality • This is not the whole story! • Perception, attention switching • Multimodality • McGurk effect • Lip reading used in noisy environments • More extensively by those with hearing problems • Visual information used, but only when appropriate • Conscious and unconscious processing • Speech often works on prediction Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

41. Multimodal Speech Processing • A different direction for listening devices and hearing aids • Consider how people actually hear • Lip reading as part of speech filtering • Cognitively inspired nuanced use of visual information Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

42. General Cognitive Framework • Taylor discussed the creation of a cognitive being • Language • Consciousness • Decision making • Memory • Emotional coding • Aim is to consider a more general purpose approach • Basal Ganglia inspired decision making • Concept based emotion analysis • Multimodal speech interpretation capabilities • Dual level processing • Can they be combined into a multimodal framework? Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

43. General Cognitive Framework • Multimodality • More environmentally aware • Additional sensors to feed into a vehicle control system • Vision, sound, weather conditions etc. • Communication • Communicate with those in the car and outside • Speech recognition and generation • Sentiment analysis from passengers • Able to learn and adapt to wishes of those in car • Adjust behaviour to suit conditions and emotions • Multimodal social and cognitive agents Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

44. Sentic Blending: Scalable Multimodal Fusion for the Continuous Interpretation of Semantics and Sentics • A general and scalable methodology termed sentic blending, for interpreting the conceptual and affective information associated with natural language through different modalities: • enables the continuous interpretation of semantics and sentics (i.e., the conceptual and affective information associated with natural language); • based on the integration of an affective common-sense knowledge base with any multimodal cognitive signal image and control processing module. • operates in a multidimensional space that enables the generation of a continuous stream characterizing user’s semantic and sentic progress over time - despite the outputs of the unimodal categorical modules having very different time-scales and output labels. • Uses decision fusion Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

45. A sample schema of continuous multimodal fusion through sentic blending Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

46. An application example: SenticNet Engine Ensemble streams obtained when applying sentic blending to the SenticNet engine (left) and the facial expression analyser (right), without ‘sentic kinematics’ filtering. Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

47. An application example: SenticNet Engine Ensemble stream obtained when applying sentic blending to the proposed conversation, with (right) and without (left) using ‘sentic kinematics’ filtering. Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory

48. Performance Comparison Confusion matrix obtained combining the five classifiers. Success rates for neutral, joy, and surprise are very high, but disgust, anger, and fear tend to be confused Confusion matrix obtained after human assessment. Success ratios considerably increase, meaning that the adopted classification strategy is consistent with human classification.

49. General Cognitive Framework • Considers the emotional states of others • Considers aspects of human cognition • Considers the issue of language • Considers benchmark problems • Convincing communication • Could be extended to include vehicle and language control • Driving, extremely challenging problem • Dual level processing • Cognitively inspired use of different modalities • Dual layer processing is unifying Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory

50. Acknowledgements • Everyone who helped to organise this conference! • All of the COSIPRA Lab • http://cosipra.cs.stir.ac.uk • Dr Erfu Yang, Prof Leslie Smith, Dr Erik Cambria Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory