Meeting Modelling

1. Meeting Modelling Rutger Rienks Dennis Reidsma, Nataša Jovanović

2. Overview • Introduce the multimodal framework • Tailor the framework to the meeting domain

3. The Multimodal Framework • We start with an unannotated corpus that should be domain representative, large enough to do relevant research and be accessible. The first step should be the identification of research goals of a project in general. In our case we want insights into human-human interaction patterns. So we need research questions on human behaviour. Tools can help answering the research questions by providing means to represent data (e.g. replay, summarize or make data browsable.) • To facilitate structured access to corpus data and reveal phenomena contained in this data two things are required : • There is a need for models and theories of human interaction to structure representations. • We need to fit or augment the data in terms of these models using layered annotations. The annotations are seperated into two distinct groups : The objective event annotations such as postures or speech recognition all dealing with the form The interpreted events dealing with the semantic meaning or function of the events. The resulting framework enables the positioning of all the on-going work on multi-modal interactions. It can be used to simplify the integration of results from research conducted at various sites. And it can be tailored to a specific domain. Research questions on human behaviour Tools Layered Annotations Semantic interpretations Models and theories of human interaction Objective events Unannotated Corpus

4. The Meeting Model 1/2 • We apply the previous model to the meeting domain and focus on the annotation aspects and interaction models. Interdependent annotation- format relations annotation formats Models of and theories of human interactions. Interpreted and combined events Recognized features: words, gaze, postures,..

5. The Meeting Model 2/2 High level views of the meeting, e.g. segmentation into meeting activities At the highest level a segmentation into meeting-activities can be derived from the lower levels. These components can be obtained using : -Fixed common sense segments (presentation, discussion, etc.) using fixed rules -Derived segments obtained from statistical models trained on occurences of lower level component combinations. Based on the lower level annotations, three distinct groups of higher level annotations can be derived from one or multiple lower level segments. - Tasks and Topics - Communicative Acts (CA’s) - Non Communicative Acts (NCA’s) To obtain these groups the models and theories for interaction descriptions are used. The communicative acts can e.g. be according to the SBWD-DAMSL or the IPA annotation schemes. On the next level we are again able to derive three different state types from the lower level components -The Argumentative states -The Emotional states -The Physical states Here again models define the labelset of the annotation schemata. Argumentative state Emotional, Interpersonal state Physical State Interpreted and Combined Events Non Communicative Acts Tasks and Topics Communicative Acts Recognized features: words, gestures, postures, nonwords, etc.

6. Conclusions • We have created a global framework for corpus based human-human interaction research • All the work in this field can be positioned within this framework • We tailored the framework to comprise the meeting domain