Institute of Systems and Robotics

TRIDENT2 - Meeting, Dec. 28th Teleconf. Meeting, Dec. 28th Presentation by Prof. Jorge Dias Institute of Systems and Robotics Institute of Systems and Robitcs http://paloma.isr.uc.pt

Institute of Systems and Robotics, ISR-UC • The Institute of Systems and Robotics, has activities in the areas of: • robotics vision, • autonomous systems, • multi-sensor fusion and integration, • tele-operation, • sensor development, • soft-control and motors and drives.

Institute of Systems and Robotics, ISR-UC PARTNER PRESENTATION: • The Institute of Systems and Robotics (ISR-UC, www.isr.uc.pt) is an associated university research institution of the University of Coimbra. ISR-UC promotes advanced multidisciplinary R&D in the areas Robotic Manipulation, Medical Robotics, Assistive Systems, Autonomous Mobile Robotics, Intelligent Transportation Systems, Computer Vision, Biomedical Engineering, Automation, Control Theory, Operations Management, Sustainable Energy Systems, and cooperates with local neurosciences groups. • ISR- UC is involved in over 25 national projects and over a dozen international projects. More recently in BACS - Bayesian Approach to Cognitive Systems (FP6-IST-027140), PROMETHEUS - Prediction and interpretation of human behaviour based on probabilistic structures and heterogeneous sensors (FP7 - 214901) and HANDLE - Developmental pathway towards autonomy and dexterity in robot in-hand manipulation (FP7-2008– 231640), both applying Bayesian learning and inference techniques, resulting in 2 completed PhD thesis and 3 under work. • ISR-UC received the outstanding grade "Excellent" as a result of the last R&D Portuguese Unit Evaluation, being the only Electrical and Computer Engineering (among a total of 25 units) to receive that distinction.

Mobile Robotics Lab • Some of the researches topics in the Lab(Most of them dealing with uncertainty using probabilistic approaches): • Multi-sensor fusion for 3D reconstruction; • Bayesian Multi-perception • Mobile Robot Sensing • Human Machine Interfaces • Human Motion Modeling • Human and Robotics Dexterous Manipulation • Human Behavior Understanding • Multi-sensor 3D Data Registration • Health Care • Visual Navigation and Tracking for Human Machine Interaction http://paloma.isr.uc.pt Some past and ongoing projects which our group have been collaborating with human data extraction, learning, Bayesian inference, 3D reconstruction, etc.:  more projects athttp://paloma.isr.uc.pt

Our Group within MRLab for TRIDENT2 • Jorge Manuel Miranda Dias born on March 7, 1960, in Coimbra, Portugal and has a Ph.D. degree on Electrical Engineering at University of Coimbra, specialisation in Control and Instrumentation, November 1994. Jorge Dias conducts his research activities at the Institute of Systems and Robotics (ISR-Instituto de Sistemas e Robótica) at University of Coimbra. Jorge Dias’ research area is Computer Vision and Robotics, with activities and contributions on the field since 1984. He has several publications on Scientific Reports, Conferences, Journals and Book Chapters. Jorge Dias teaches several engineering courses at the Electrical Engineering and Computer Science Department, Faculty of Science and Technology, University of Coimbra. He is responsible for courses on Computer Vision, Robotics, Industrial Automation, Microprocessors and Digital Systems. He is also responsible for the supervision of Master and Ph.D. students on the field of Computer Vision and Robotics. Dr. Jorge Dias Head of the Lab • Jorge Lobo (Jorge Nuno de Almeida e Sousa Almada Lobo) was born on the 23rd of September 1971, in Cambridge, UK. In 1995, he completed his five year course in Electrical Engineering at Coimbra University. In April 2002, he received the M.Sc degree, and in June 2007 he received the Ph.D degree from the University of Coimbra. He was a junior teacher in the Computer Science Department of the Coimbra Polytechnic School, and later joined the Electrical and Computer Engineering Department of the Faculty of Science and Technology at the University of Coimbra, where he currently works as Assistant Professor. He is responsible for courses on Digital Design, Microprocessors and Computer Architecture. His current research is carried out at the Institute of Systems and Robotics, University of Coimbra, working in the field of computer vision, sensor fusion, and mobile robotics. Current research interests focus on inertial sensor data integration in computer vision systems, Bayesian models for multimodal perception of 3D structure and motion, and real-time performance using GPUs and reconfigurable hardware. He has participated in several national and European projects, most recently in BACS, Bayesian Approach to Cognitive Systems, and HANDLE. Dr. Jorge Lobo

Our Group within MRLab for TRIDENT2 • Rui Rocha was born on 13 May 1973, in Castelo de Paiva, north of Portugal. He completed his Electrical and Computer Engineering degree (specialization on Automation, Control and Instrumentation) on July 1996, his M.Sc. degree (specialization on Industrial Informatics) on March 1999, and his Ph.D. degree on May 2006, all by the Faculty of Engineering of the University of Porto. Between February 2000 and May 2006, he was a Teaching Assistant at the Department of Electrical and Computer Engineering, in the Faculty of Sciences and Technology of the University of Coimbra. • Currently he is an Assistant Professor at the Department of Electrical and Computer Engineering and a researcher at the Institute of Systems and Robotics, in the Faculty of Sciences and Technology University of Coimbra. • His main research topics are cooperative multi-robot systems, 3-D map building, distributed architectures and Intelligent Transportation Systems. Dr. Rui Rocha • Paulo Menezes is an Assistant Professor at the Department of Electrical and Computer Engineering of the Faculty of Sciences and Technology of the University of Coimbra, Portugal. He received his Ph.D. degree from the University of Coimbra for his dissertation on "Multi-Cue Visual Tracking for Human-Robot Interaction." He also holds M.S. and B.S. degrees from the University of Coimbra in Electrical Engineering.He is a senior researcher of the Institute of Systems and Robotics and belongs to the Mobile Robotics Laboratory team. • His research interests are: robotics, computer vision, human-robot interaction, augmented reality and new technologies for health care and life quality support. He is involved in several European and National funded projects on these fields. Dr. Paulo Menezes

Our Group within MRLab for TRIDENT2 • Diego Resende Faria was born on Aug.17th , 1979 in Londrina (state of Parana), Brazil. He is a Ph.D. student at the University of Coimbra, Portugal. He is Researcher at the Institute of Systems and Robotics - Department of Electrical and Computer Engineering - University of Coimbra. He is under the supervision of Prof. Jorge Dias (advisor) and Prof. Jorge Lobo (co-advisor). He is sponsored by a Ph.D. scholarship from the Portuguese Foundation for Technology and Sciences. • He has graduated in Information Systems Technology in 2000 and has finished a Computer Science Specialisation Course in 2002 at the State University of Londrina, Brazil. He holds an M.Sc. degree in Computer Science from the Federal University of Parana, Brazil, since 2005. • Currently, Diego Faria is collaborating as researcher on the European Project HANDLE within the 7º framework FP7. His research interest is Robotic Grasping, Multimodal Perception, Imitation Learning, Computer Vision and Pattern Recognition. Diego Faria • Ricardo Filipe Alves Martins was born on the 15th of October 1984 in Proença-a-Nova, Portugal. Ricardo has an M.Sc. degree in Biomedical Engineering from the University of Coimbra, Portugal obtained in 2008. Currently, he is a Ph.D. student and researcher at the Institute of Systems and Robotics, Department of Electrical Engineering and Computers, University of Coimbra, Portugal. He is sponsored by a Ph.D. scholarship from the Portuguese Foundation for Technology and Sciences. He is collaborating as researcher on the European Project HANDLE within the 7° framework FP7. His research interests are Robotic Grasping/Haptics, Multimodal Perception and Imitation Learning. Ricardo Martins

TRIDENT Project overview • PHASE I (Survey) -The Autonomous Surface Craft (ASC) is launched to carry the Intervention Autonomous Underwater Vehicle (I-AUV) towards the area to be surveyed. -Then, the I-AUV is deployed (1) and both vehicles start a coordinated survey path (2) to explore the area. -The ASC/I-AUV team gathers navigation data for geo-referencing the measurements (seafloor images and multibeam bathymetry profiles). -Finally, the I-AUV surfaces (3) and contacts to the end user to set-up and acoustic/optical map of the surveyed area. -Using this map, the en user selects a target object (an object of interest) as well as a suitable intervention task (grasping, hooking, etc...). - ISR-UC potential contribution on this topic

TRIDENT Project overview • PHASE II (Intervention) -After selecting the target, the ASC/I-AUV team navigates towards the target position. Then, the ASC performs dynamic position (4) while keeping the I-AUV inside the USBL cone of coverage. -Then, the I-AUV performs a search (5) looking for the Target of Interest (ToI). When the object appears in the robot field of view, it is identified and the I-AUV switches to free floating mode using its robotic arm as well as the dexterous hand to do the smart manipulation (6). -Finally (7), the I-AUV docks to the ASC before recovery. - ISR-UC potential contribution on this topic

Timeline -->future Can be used as input for Learning Models (TRIDENT-2)

Scientific Objectives (SO) & Technical Objectives (TO) Description

Scientific Objectives (SO) & Technical Objectives (TO) Scenarios

TRIDENT2: Example of Application Phase I - TO1 • Motivation • Both vehicles are deployed in the mission environment; • Collaborative exploration strategy to map the unknown environment; • Approach • Adapt the collaborative trained exploration strategies parameters to unknown environment contexts; • Simulation environment – Learning phase • Environment context A –>Demo: AUV1 & AUV2 exploration strategy parameters A • Environment context B–>Demo: AUV1 & AUV2 exploration strategy parameters B • Real world experimental environment– Inference phase • Environment context analysis • -Description of the perceived environment context as a probabilistic combination of previously known/learned environment contexts. • Estimation of the ponderated exploration strategy parameters of the new unknown environmental context. 2/29

TRIDENT2: Example of Application Phase II - T02, T03, T04, T05 • Flow chart that encloses two cases: • the first one is a known context where the object (target) is known a priori • Imitation of previous knowledge (trajectories) taking into consideration the object pose (to generate hypothesis of possible grasps that influences also the trajectories; • the second, the object is not known (during exploration, rocks/stones, tubes, object that was not used in the learning phase) • *in this case a generalization process by similarities is used. The object is segmented in such way that is possible to generate known shapes (primitives) that composes the object, so that the grasp planning can be applied to that shape, including the trajectories. * Example of similarities: Given a unknown object (not learned before), then its shape is decomposed in primitives (superquadrics) to approximate the shape into known geometric shapes to be possible to generate a specific grasp for some of the segmented parts: Known geometrical shapes and its pose can generates known (learned) grasps and trajectories

TRIDENT2: Example of Application Phase III – TO6 • Motivation • Both vehicles working together and under coordination with the two robotic arms to assemble multi-elements objects; • Approach • Cooperative manipulation movements and assembly strategy described at a symbolic kevel • Simulation environment – Learning phase • Multi-elements object configuration A  Object assembly demo 1, 2 ... N • Extraction of the finite set of primitives – task dictionary • -Characterization of each primitive using lower level signals (trajectories, control signals, …) • Extraction of a generalized task structure (general primitive sequence) • -Extraction of some of the primitive sequence rules (required precedence, …) - grammar rules of the task primitives dictionary • Real world experimental environment– Inference phase • Multi-element object configuration recognition Primitive sequence generation 2/29

TRIDENT2: Generalization

Open Issues for Discussion • What are the signals provided by the other project modules that can be used for learning in T01, T02, T03 ? : • Combined trajectories of the Arm and UAV ? • Combined trajectories/pose of object, the gripper, the Arm and UAV? • Controls signals and trajectories accomplished in the demonstrations ? • What are the object detection features provided by the environment mapping module? • Target object tracking - 2D and 3D cases? • Object Detection (template matching, shape approximation) – Known and unknown objects? • Automatic Segmentation (floor, 3D object map) ? • Intervention for 3D case (already tested?) 2/29

TRIDENT2 - Meeting, Dec. 28th Thank You! END. Institute of Systems and Robotics Institute of Systems and Robitcs http://paloma.isr.uc.pt

Institute of Systems and Robotics