Multi-modal Interfaces

1. Multi-modal Interfaces “The focus of IST in FP6 is on the future generation of technologies in which computers and networks will be integrated into the everyday environment, rendering accessible a multitude of services and applications through easy-to-use human interfaces“ IST WP 2005-2006

2. Our Mission The challenge is: • To make interfaces as simple as possible, and interaction as easy as possible • For citizens to be able to access, receive and use information in their own language We support the research, development and integration of: • Advanced technologies for interfaces & interaction • Integrated multilingual information systems and services We focus on: • Technology-mediated interaction and communication between people, between their devices, and between people and content • Combining traditional human language services, like human translation and interpretation, and advanced language technologies and resources E1 Interfaces

3. The Facts Call Title: IST Call 5 Call Identifier: FP6-2004-IST-5 Strategic Objective: 2.5.7 Multi-modal Interfaces Open: 17 May 2005 Close: 21 September 2005 at 17.00 (Brussels local time) Budget: 54 M€ Instruments: IP’s, STReP’s New/Traditional Instruments: 60/40

4. Main challenges • Need to provide information artefacts to be used by increasingly diverse user groups • Expanding context of use, from scientific and business use to residential and nomadic use • Increasing variety of media and devices used to access a community-wide pool of services and information resources • Demonstrate added-value of multi-modality and performance advances through multi-modal interaction • Go beyond system integration for demonstration purposes and target real world solutions • Emphasise multi-user scenarios and non-business applications • Stronger focus on defining the expected impact • Need to promote cooperation between small European companies • Promote research that could lead to spin-offs

5. An R&D agenda • Key interaction technologies (speech, natural language input, haptics, vision, sensors, gesture, agents, …, including unconventional modalities) and their modelling and integration into interaction platforms • Interface architectures, tools for cost effective integration, e.g. fusion, and toolkit interoperability • Computational models of multi-modality, and advanced corpora for research • Re-develop a serious effort on automatic translation • Reinforce work on robustness and acceptability of speech recognition and research on speaker-independent software • New learning paradigms in (spoken) language translation and portability to new languages, including greater focus on language resources as computer processable and reusable resources • Design processes, methods and tools that support scalability and modality independence • Empirical science base for understanding end-users, e.g. computational models, test suites and benchmarks

6. Multi-modal Interfaces Objective: natural and easy to use interfaces that use several modalities or are multilingual • Natural interaction between humans and the physical or virtual environment • Multilingual communication systems IST WP 2005-2006

7. Multi-modal Interfaces Natural interaction between humans and the physical or virtual environment • Interfaces that are autonomous, and learn and adapt to user intentions and behaviour, in dynamically changing environments • Featuring unconstrained, robust interaction, recognise user reactions and respond to them in a natural way • Selecting the right combinations of modalities according to the preferences and context • Looking at both the fusion of information related to different modalities and their channelling to multiple modalities IST WP 2005-2006

8. Multi-modal Interfaces Multilingual communication systems • For unrestricted domains, including task-oriented, real-time understanding of spontaneous spoken and gesture input • Address novel learning paradigms that exploit contextual information, human and linguistic knowledge in a more effective way than currently done • Portability of new languages taking advantage of methods and techniques developed for languages already covered is a further challenge to be addressed, e.g. in the context of new EU languages IST WP 2005-2006

9. Multi-modal Interfaces This covers: • User modelling, system design, visual recognition and tracking, language understanding and spoken language translation • Applications should focus on proof of concepts • Includes consumer, nomadic, creative, artistic and gaming applications IP - system-level objectives in natural interaction and multilingual communication STReP – can also include work on language understanding and spoken language translation IST WP 2005-2006

10. Instruments • Integrated Project (IP): • Result driven with strong industry/academic consortium and users well integrated • Focus on impact, exploitation, and replication • Multi-faceted with R&D, demonstration, training, awareness building, standardisation, … • 10-15 partners, 5+ countries, 4 years, 5-10 M€ • Targeted Project (STReP): • Problem oriented with the right team for the S&T challenge • Single-problem R&D with high innovation/risk/reward • Investigating new research options, proof of concept, or competitive approaches • 5-8 partners, 4+ countries, 3 years, 2-4 M€ draft

11. The Evaluation Criteria • Relevance: • Clarity – make it a clear yes/no decision • Reference national and European foresight results • Quality of consortium: • Tell them what your principle qualities are – quality of R&D, industrial participation, exploitation potential, SME involvement, uniqueness of consortium, track record, … • Well balanced in terms of expertise, effort and exploitation • All partners must have substantial and well defined roles to play (no window dressing, no hangers on) • Users and end-users are present and have clearly defined roles • Resources: • Clear task allocation and no unnecessary overlaps • Justify costs, effort allocation, and link explicitly to results draft

12. The Evaluation Criteria • Management and organisation: • Clear work-plan and good progress indicators and tracking are key signs • Indicate some valuable intermediate milestones • Indicate how risks will be assessed and what are you back-up options or contingency plans • How decisions are made, and can they be made in a timely way – conflict resolution and consensus building • Good interlinking between objectives, activities and work-packages (Gantt) • Exploitation/dissemination: • Protect and manage IPR and provide access to partners • Start actions early and be precise about options • Go beyond just “publishing results” – make an impact and above all be visible draft

13. The Evaluation Criteria • European added-value: • Show added value to existing national and European programmes • Compliment ongoing efforts of most partners • S&T excellence: • Position the project vis-à-vis wider strategic objectives • Define clear challenging but attainable objectives • Describe state-of-the-art and go beyond (no catch up) • Commit to some key deliverables/impact measures • Justify balance between research and development • Stress visible industry-scale validation through experiments, prototypes, demonstrators… • Show that you can scale to industrialisation • Don’t just be interesting, be manufacturable draft

14. Remember… Its all about generating, demonstrating and validating new knowledge Be ambitious, but clearly define your scientific and technological objectives – think 2010+ Be unique, innovate but be industrially relevant - include key industry players and ensure broad industrial impact Be evaluator friendly, make it a compelling read, easy to understand, a give them a reason to remember it at the end of the week Be credible, one proposal, no “template” proposals, do what is needed to get the result, not more, but not less – remember 54 M€ represents ~1,200 person-years of effort Check your ideas with us!

15. Further information General FP6: http://europa.eu.int/comm/research/fp6/ http://www.cordis.lu/ IST: http://www.cordis.lu/ist IST infodesk email: ist@cec.eu.int fax: +32 2 296 83 88 Bernard Smith email: bernard.smith@cec.eu.int