Cécile Huet – cecile.huet@ec.europa.eu Unit E5 - Cognitive Systems, Interaction, Robotics

1. The Strategic Role of Vision in Cognitive Systems and Robotics - A Perspective from the EU Research ProgrammeECCV 2010, Special session on "Research Funding for Vision", September 9, 2010 Cécile Huet – cecile.huet@ec.europa.eu Unit E5 - Cognitive Systems, Interaction, Robotics DG Information Society and MediaEuropean Commission http://www.cognitivesystems.eu

2. Outline • History • Projects portfolio • Perspectives and future opportunities • Conclusion

3. It all started with VISION:“Cognitive Vision” Initiative in 1998 (FP5): Rationale • Limitation of computer vision systems: • Lack of ROBUSTNESS/VERSATILITY/SPEED • Cognitive vision • combine state-of-the-art image processing with cognitive reasoning: abstract representation, categorisation, memory organisation, self-adaptivity, contextual knowledge, behavioural/scene interpretation,…

4. Projects portfolio • >140 projects - €480 million EU funding. • FP5 (1998-2002) Cognitive vision - 9 projects • FP6 (2002-2006) Cognitive Systems + Advanced robotics - 45 projects • FP7 (2007-2013) Cognitive systems, Interaction, Robotics 68 running projects + 22 projects under negotiation

5. FP5 Cognitive Vision - achievements • FP5 – DETECT Real Time Detection of Motion Picture Content in Live Broadcasts -> detect/track brand appearance in sport videos

6. FP5 Cognitive Vision - achievements • FP5 – ACTIPRET Interpreting and Understanding Activities of Expert Operators for Teaching and Education Recognition of hand and arm gestures: original image with overlay and 3D representation Long term goal: giving hints on how to clear the paper jam in a copy machine

7. FP5 Cognitive Vision - achievements • LAVA - Learning for adaptable visual assistants • Categorization and interpretation of large numbers of objects, scenes and events, in real settings. • Recognition/Categorization: Occlusion – cluttered background – viewpoint-rotation-scale,… invariance, lighting conditions,… • Categorization - Cope with different forms of the same object (shape, colours, size, angle etc) and an open set of instances of the object • Scene understanding – disambiguation –> multi cue/information fusion – context,… Source: www

8. FP6: Cognitive systems research - motivation Beyond vision…. Achieving naturalness, versatility, robustness at system level • from constrained toreal-worldenvironments • from application-specific to general solutions • from component methods to a systems approach • from a ‘monodisciplinary’ research effort to working across disciplines RESULTS: Integration of perception (attention mechanisms, affordances, multi-sensory/cue…), representation, learning, reasoning, decision-making, action and communication, cooperation (HRI/RRI), anticipation,… Strong multidisciplinarity (neurosc., cog sci., engineering, AI,…)

9. FP6: Cognitive systems projects - classification • Robotics (navigation – manipulation) - action-perception(COSPAL-SPARK), affordances (MACS-PACO-PLUS), anticipation(MINDRACES), reasoning - understanding (COSY), attention, knowledge-reasoning (GNOSYS), observation (MATHESIS), decision-making (DECISIONS-IN-MOTION), reasoning & self-preservation (ICEA) , Joint Action (JAST), Active haptic(SENSOPAC), developmental (ROBOT-CUB) • Interfaces - audio-visual- HHI/HRI(POP), HRI (PACO-PLUS, COSY), HRI-RRI (JAST) – ECAS (RASCALLI) • Assistive technologies - Blind (DECISIONS-IN-MOTION- CASBLIP) – driving assistance (BACS) – Augmented human action (PACO-PLUS) • Scene analysis/digital content - audio-visual + attention (POP-DIRAC), language & information discovery (RASCALLI), description image/video + text (CLASS), man-made scenes interpretation (eTrims), description of human behaviour(movement-facial expression) from video sequences (HERMES) • + euCognition / PASCAL

10. FP6: Advanced Robotics projects - classification HRI • FEELIX GROWING: FEEL, Interact, eXpress: a Global appRoach to develOpment With INterdisciplinary Grounding • INDIGO: Interaction with Personality and Dialogue Enabled Robots • COMMROB: Advanced Behaviour and High-level Multimodal Communication With and Among Robots • PHRIENDS: Physical Human-Robot Interaction: DepENDability and Safety ROBOT – FOR HUMAN ASSISTANCE (environments: Firefighter/Urban pedestrian area/home) • VIEWFINDER: Vision and Chemiresistor Equipped Web-connected Finding Robots / GUARDIANS • URUS: Ubiquitous Networking Robotics in Urban Settings • Robots@home: An Open Platform for Home Robotics (platform-perception-navigation) SWARM – for human assistance or autonomous tasks • IWARD: Intelligent Robot Swarm for Attendance, Recognition, Cleaning and Delivery (+HRI) • ROBOSWARM: Knowledge Environment for Interacting ROBOt SWARMs (IT infrastr./Cleaning Hospital) • GUARDIANS: Group of Unmanned Assistant Robots Deployed In Aggregative Navigation supported by Scent detection Network of Excellence / Coordination and Support Actions • EURON: European Robotics Research Network • CARE: Coordination Actions for Robotics in Europe • RoSta: Robot Standards and Reference Architectures (+benchmarking) • RAWSEEDS: Robotics Advancement through Web-publishing of Data Sets

11. FP6 – Vision: Examples • CASBLIP: Cognitive Aid System for Blind People • Sensory substitution: from vision to audio information • DECISIONS-IN-MOTION: Neural Decision-Making in Motion • understand the neural mechanisms underlying fast target selection and sensory-guided decisions and actions in humans in motion • transfer this knowledge into an artificial visual system for automated robotic navigation.

12. Challenges greatly improving robustness etc. requires rethinking the way systems are engineered -> theories are needed Scientific approach: understandingof what both natural and artificial systems can and cannot do, and how and why integration of disciplines: robotics, AI, computer vision, natural language, cognitive science, psychology,… mathematics,… philosophy of mind,… Equally importantly we need to be able to support claims that progress is being made identify real applications that would greatly benefit from such systems FP7 - Challenge 2: Cognitive Systems, Robotics, Interaction

13. VISION Cognition & Language emergence in robots FP7 – CALL1 – PROJECTS CLASSIFICATION Perception & Sensors Dexterous Manipulation Cognition for Human Assistance ALEAR ITALK ROSSI Medical – Rehab./Surgery GRASP CHIROPING DIPLECS ROBOCAST MIMICS CHRIS DEXMART EYESHOTS CogX SEMAINE PINVIEW LIREC HRI Multimodal interaction SEARISE PROMETHEUS CoFRIEND SCOVIS POETICON SF SPARK II REPLICATOR CLASSIC EMIME HMI Autonomous Robots Scene understanding Sensor network Spoken language PASCAL 2 Machine Learning

14. FP7 – CALL3 – PROJECTS CLASSIFICATION: SCENE UNDERSTANDING MULTI-ROBOT / SWARM UNDERWATER AERIAL URBAN Co3AUVs SHOAL FILOSE EUROPA SFLY SCANDLE ACOUSTIC SCENE ANALYSIS NAVIGATION HRI SCENE UNDERSTANDING CA in Cog Sys EUCOGII

15. VISION FP7 – CALL3 – PROJECTS CLASSIFICATION FROM AN APPLICATION ORIENTED PERSPECTIVE INDUSTRY / MANUFACTURING Co3AUVs SCANDLE MONITORING SURVEILLANCE SECURITY FILOSE SHOAL SFLY BRICS ROSETTA SEARCH & RESCUE ENVIRONMENTAL MONITORING IM_CLEVER HANDLE ECHORD ECCEROBOT MOBILE MANIPULATION SOCIAL-COMMUNICATIVE SKILLS (multimodal interaction – TV-Host) MANIPULATION SORTING OBJECTS ASSEMBLY PROSTHETICS STIFF EUROPA HUMANOBS SERVICE ROBOTICS & HRI ROBOSKIN HUMOUR GUIDANCE DELIVERY TRANSPORT REHABILITATION MOTOR SKILL LEARNING PROGRAMMING BY DEMO/ TOUCH-BASED SOCIAL INTERACTION

16. BIO-INSPIRED LOCOMOTION MOTOR CONTROL MULTIMODAL LANGUAGE HAPTICS + VISION ROBOSOM IURO RADHAR DEXTEROUS MANIPULATION NIFTI AMARSI THE AIROBOTS ALIZ-e HUMAVIPS TRIDENT GERT COGNITO FIRST-MM PERCEPTION: VISION ROBOEARTH GARNICS VISION TACO AUDIO- VISUAL MASH HRI NAVIGATION VANAHEIM WWW SHARED DB: Robots actions euROBOTICS SENSOR NETWORK/ HUMAN ACTION RECO CA in ROBOTICS FP7 – CALL4 – PROJECTS CLASSIFICATION

17. Shared resources: Contribute to and benefit from:RoboEarth – Robots sharing a knowledge base for world modelling and learning of actions - http://www.roboearth.org/ • World Wide Web for robots: a giant network and database repository where robots around the world access and continually update to share information and learn from each other about their behaviour and their environment. • Improve any robot’s 3D sensing, acting and learning capabilities.

18. Shared resources: Contribute to and benefit from:MASH - Massive Sets of Heuristics for Machine Learning • Design of complex learning systems integrating large and heterogeneous families of feature extraction modules. • Open web-based platform where contributors will find tools to interact and communicate with each other, and the means to integrate their feature extractors in continuously running experiments. • Quantitative evaluation of submitted modules • Performance will be measured on action selection with a robotic arm and in a simulated 3d environment, and object detection. • http://mash-project.eu/

19. HRI + HR COOP.: WORKER DETECTION + INTENTION RECO- SKILL LEARNING PACKAGING ELECTRONIC CONSUMER GOODS PACKAGING CustomPacker CUSTOMPACKER TAPAS NAVIGATION / PLANNING: MOBILE MANIPULATION / MOBILE ASSISANT: LOGISTICS + ASSEMBLY ROBOFOOT VISION MANIPULATION/ HANDLING SORTING OBJECTS VISUAL INSPECTION VISUAL SERVOING SHOE FP7 – CALL FoF – Smart Factories: ICT for agile and environmentally friendly manufacturing – Target outcome c) Robotics-enabled production processes tested and validated in real-world environment PRECISION FLEXIBILITY/SCALABILITY RELIABILITY ROBUSTNESS VERSATILITY MANIPULATION NON-RIGID / HEAVY OBJECTS

20. DEXTEROUS MANIPULATION DEXTEROUS MANIPULATION NAVIGATION HRI/SOCIAL INTERACT GOAL-DIRECTED/ANTICIPATION VISION + Object/scene understanding SENSOR NETWORK FP7 – CALL6 – PROJECTS CLASSIFICATION (! Subject to successful negotiation) CA in ACS

21. What next?The 2010 Call for Proposals – FP7 - ICT Call 7 - http://www.cognitivesystems.eu INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED • TARGET a):Research projects on Robotic systems operating in real-world environments: + functionalities + autonomy, safety, robustness, efficiency, and ease of use AS APPROPRIATE: + new materials and advanced sensor, actuator, effector, memory and control technologies. ->> validation through REAL-WORLD scenarios • TYPE OF PROJECTS: STReP (Specific Targeted Research Projects) High risk/Focussing on specific research topic/component IP (Integrated Projects) System oriented • BUDGET: 70M€ • DEADLINE: 18.01.2011

22. What next?The 2010 Call for Proposals – FP7 - ICT Call 7 - http://www.cognitivesystems.eu INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED • TARGET d) Coordinated Action(CA) Fostering communication and co-operation between robotics and cognitive systems research communities: + knowledge sharing: EU, national, and international; + open source; R&D roadmaps, market potential, user acceptance, standardisation, education, ethics, socioeconomic impacts; outreach to relevant professional and general audiences. • TYPE OF PROJECTS: CA • BUDGET: 3M€ • DEADLINE: 18.01.2011

23. What next?The 2011/12 Call for Proposals – FP7 - ICT Call 9 - http://www.cognitivesystems.eu INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED • Cognition and control in complex systems (IP and STReP) • Cross-fertilisation between academic and industrial robotics research (IP) • Targeted competitions for smarter robots (CA) • INDICATIVE DEADLINE: 17.04.2012

24. Perspectives • Outcome so far and next steps • New research questions -> need for further scientific progress • Also technological progress and application motivated research, building on previous results – innovation in integration and implementation of results to improve robotics functionalities and features. • Real world scenarios: source of research questions + validate research results • Demonstrate progress: Benchmarking

25. CONCLUSIONFP7 – Workprogramme text « Research and development […] will be guided by demanding, yet pragmatic application scenarios » Role of vision?

26. CONCLUSIONFP7 – Workprogramme text « Research and development […] will be guided by demanding, yet pragmatic application scenarios » Critical!!

27. THANK YOU! http://www.cognitivesystems.eu