ICT Networking Session « SEMANTIC ADAPTATION IN AFFECTIVE INTERACTION »

1. ICT Networking Session«SEMANTIC ADAPTATION IN AFFECTIVE INTERACTION» STEFANOS KOLLIAS National Technical University of AthensComputer Science Division School of Electrical and Computer Engineering Lyon, France, November 25, 2008

2. Introduction • Affective computing is of major importance in Human Machine Interaction; it involves perception, interpretation, cognition, expression. • Related International conferences include ACII2005, ACII2007, LREC2007, LREC2008, ACII2009. • EU IST projects and networks (Ermis, Interface, Safira, Humaine, Callas, Semaine, Feelix-Growing, Metabo) investigate(d) different issues of affective computing: theories and models of emotional processes, computational modelling, emotional databases, input signal analysis, emotion recognition, generation of embodied conversational agents.

3. State-of-the-art Affective computing systems: • model and analyse single or multi-modal affective cues; • extract and use statistical information and rules; • create data sets and environments to perform user state detection; • include Embodied Conversational Agent (ECA) synthesis and interaction.

4. Requirements and achievements • Common frameworks: i) discrete, dimensional, component, appraisal emotional & affective models, ii) cognitive & goal-based interaction computational models iii) Facial Action Coding System (FACS) and MPEG-4 models for facial, viseme and body motion analysis and synthesis, iv) features extracted from paralinguistic speech analysis, v) affect indicating words from linguistic speech recognition, vi) physiological characteristics and cues, vii) multi-modal integration through feature or decision fusion. • Need to address emerging requirements: humanlike interactions, less constrained environments, adaptive artificial systems

5. Problems encountered • Synchronisation, contradiction, co-existence lack, evolution. • Fusion of different signals and cues requires inferring on contradictory or ambiguous emotional cues • Assuring that interaction is acceptable and appropriate in terms of user experiences. Lack of common knowledge framework consistently representing taxonomies, rules and correlations in affective computing. • Recognition performance and fusion of multi-modal inputs is not satisfactory across varying and different environments, • Lack of interaction cycles where, in real time, the analysis loop feeds the synthetic one with info which is then reused further through user’s affective feedback, • Including the cognitive component and the context of interaction in the loop makes the situation much harder and systems almost impossible to benchmark.

6. Problems encountered (2) • Test databases utilize sets of well-defined multi-modal (aural, visual, biosignal) data, captured in controlled or partly uncontrolled environments, depicting humans engaged in predetermined tasks. • Moreover, knowledge obtained from one specific dataset may only be efficiently reused in the same environment. This refers to: • low-level requirements, e.g., similar lighting conditions or absence of occlusion in visual signal analysis, • user-oriented restrictions, e.g., same subjects with same physical characteristics and similar expressivities, • similar cognitive or interaction tasks, e.g. reading or speaking to an artificial listener, classifying hand gestures in predefined categories.

7. SAFE Concept • Generate a common adaptable semantic representation framework, i.e., a consistent environment to serve as knowledge substrate for affective interaction in real, uncertain, environments. • Basis: • state-of-the-art imperfect (uncertain, incomplete, vague, inconsistent) knowledge representation formalisms, • novel alignment, mapping and reasoning tools, • semantic adaptation and learning.

8. SAFE Concept (2) • Generate a learning, evolving and adapting cognitive model • Start with basic knowledge about the nature of possible interactions, users and the environment, • Include powerful sensing and reasoning mechanisms, along with the ability to infer from expert knowledge reinforced by accumulated experiences, • Result in a system gradually evolving its knowledge to incorporate its observations along with its own or the user’s evaluation.

9. SAFE Model

10. SAFE Technologies • Formal Knowledge It stores the terminology, axioms, assertions, constrains that describe affective interaction. - HCI Ontologies module (formal ontological description representing the concepts and relationships of the field, providing formal definitions and axioms that hold in every HCI environment. * abstract descriptions of users’ affective states * low level descriptions of multimodal users’ info * context description (user expressivity, goals of interaction, environmental characteristics).

11. SAFE Technologies (2) • Real environments cause inconsistencies in the Formal Knowledge • For example, the personality and expressivity of the specific user make some of the axioms and constraints of the HCI Ontology non-applicable or even wrong, according to logical entailments or user feedback. • These inconsistencies make the formal use of knowledge that the SAFE Reasoner provides rather problematic. • How to solve:

12. SAFE Technologies (3) • The reasoner detects the inconsistency • It follows a paraconsistent reasoning approach taking into account that the interaction with the user is assumed to be continuous and the user feedback will provide the system with additional important information needed in order to resolve the inconsistency. • The Knowledge Adaptation component of SAFE resolves the inconsistency through a recursive learning process.

13. SAFE Technologies (4) • Knowledge adaptation It determines the minimal set(s) of axioms that cause the inconsistency, - Inconsistency Handling module The minimal sets get represented in connectionist models and, with the aid of learning algorithms, are adapted and then re-inserted in the knowledge base. - Some parts of the knowledge represent properties of objects that are rigid, while others are highly dynamic. For the latter, the adaptation will be performed more drastically, with high learning rates, while for the former a more careful strategy will be followed.

14. SAFE Target • Semantic adaptation in affective interaction. • Key words are learning and knowledge. • Successfully combine these main components of cognition and machine intelligence. • Adaptation and evolution of ontological knowledge to effectively handle the context of interactions, i.e., specific user characteristics, goals & behaviours, or environmental changes.

15. SAFE Target (2) • The SAFE accomplishments will follow and closely relate to current state-of-the-art developments: • in W3C (RIF, OWL Development groups, EMOXG, MMI, Emotion Incubator) • in the Humaine Association (http://emotion-research.net/association), so that the SAFE system and technologies can be easily distributed and shared by the R&D community for affective interaction.

16. Advancing the State-of-the-Art • Computational models of Affect Taxonomies of affective user states, related to computational models of affect and emotions (Scherer’s proposal for distinguishing classes of affective states: * Emotions (e.g., angry, sad, joyful, fearful, ashamed, proud, elated, desperate) * Moods (e.g., cheerful, gloomy, irritable, listless, depressed, buoyant) * Interpersonal stances (e.g., distant, cold, warm, supportive, contemptuous) * Preferences/ Attitudes (e.g., liking, loving, hating, valuing, desiring) * Affect dispositions (e.g., nervous, anxious, reckless, morose, hostile)

17. Advancing the State-of-the-Art (2) • Interpretation of Affective User Behaviours • Context Modelling • Imperfect Knowledge Representation and Reasoning • Connectionist model for ontology adaptation

18. Applications • Robotics (Emotion aware, Knowledge Aggregation, Context Analysis): FEELIX-GROWING • Human Computer Interaction in Emotional Environments (Knowledge and Emotion) : CALLAS • Analysis of status of children/students in e-learning Environments (Behaviour Analysis, Knowledge of User States): AGENT-DYSL • Analysis of status of car driver (monitoring safety): METABO

19. Prospects • Requirement for Novel Cognitive Systems interweaving Knowledge & Learning/Adaptation Technologies • Theoretical and Technological Advancing of the State-of-the-art • Novel Applications: Emotion, Context and Knowledge Aware Robots, Tutors, Systems,… • New FP7 ICT Call for funding.

20. Thank you for your attention. contact details: stefanos@cs.ntua.gr