Control and Representation Vijay Kumar University of Pennsylvania

1. A M M W O R K S H O P Control and RepresentationVijay KumarUniversity of Pennsylvania • John Hollerbach • Oussama Khatib • Vijay Kumar • Al Rizzi • Daniela Rus NSF/NASA AMM Workshop March 10-11, 2005 Houston.

2. Outline • State-of-art • Historical perspective (nostalgic memories) • Accomplishments in robot control • Summary of last 21 years (WTEC study) • Recent, specific contributions (somewhat biased) • Challenges • Panelists • Discussion • What are the intellectual problem areas we should address? Infrastructure? Can we can rally around these?

3. Mobility & Manipulation 1961 Unimate Historical Perspective • 40+ years of industrial robotics • >20 years of robotics as an academic discipline • ~13 years of mobile manipulation General Motors 40 years of industrial robotics Rus Sarcos ARC Hollerbach

4. The Real Agenda for AMM • Haptics • John Hollerbach • Humanoids • Oussama Khatib • Perception/Action • Al Rizzi • Distributed/Modular • Daniela Rus • Mobility • Unstructured environments • Manipulation • Physical interaction with the environment • Closely coupled perception/action • Not physically grounded • Dynamics is important • Autonomy • Teleoperation (and therefore haptics) • Supervised Autonomy • Autonomy

5. WSJ, 3/7 “…teleoperation with time delays is a vexing problem in robotics…” “…because of the lag, it’s inevitable that the human operator will make tiny errors - errors that will in turn cascade into much bigger ones…” Robotics in the news this week

6. Literature • Domain • ~8-10% manipulation • ~3-4% grasping • ~30-35% mobility Remaining are on medical, manufacturing, industrial, sensor or “methodology” • Control/representation • Model based (~15%) • Data driven approaches (~5%) Counted papers relevant to manipulation and mobility Disclaimer: This is not a scientific study! Conferences surveyed: ICRA 1984-86, 1998-2004

7. (40%) (4%) (40%) (3 %) Literature (Compared to 1984) • Domain • ~10% manipulation • ~4% grasping • ~35% mobility Remaining are on medical, manufacturing, industrial, sensor or “methodology” Total number of papers = 74 • Control/representation • Model based (~15%) • Data driven approaches (~5%) Counted papers relevant to manipulation and mobility ~9880 ICRA papers to date Disclaimer: This is not a scientific study! Conferences surveyed: ICRA 1984-86, 1998-2004

8. Major Advances Academic/Government Labs • Inverse dynamics: application of feedback linearization to serial robots, now routinely used in industrial manipulators (e.g., ABB) • Time optimal control: along a path subject to dynamics, velocity and acceleration constraints, also used in industrial manipulators • Adaptive robot control: model based adaptive control with global stability guarantee • Nonholonomic control: control using time varying feedback or cyclic input, application of differential flat system theory, mostly applied to mobile robots and under-actuated robots. !!! !!! !? !!! Disclaimer: Not a survey of accomplishments/needs for AMM [Wen and Maciejewski, 04]

9. Major Advances (Cont.) • Flexible joint robot modeling and control: Application of feedback linearization to flexible joint robots, applied to some industrial arms. • Teleoperation: wave variable based control for delay robustness. Guarantee stability, but user would feel delayed response. • Order N simulation: Application of order N computation to forward and inverse dynamics. Essential for large number degrees of freedom, e.g., robot with flexible link, micro-robots. • Hybrid force/position, impedance control: Simultaneous regulation of motion and force, applied to machining, assembly, haptic feedback, multi-finger control ?! ! !!! !!!

10. Saturation of the area? All problems solved Not interesting Not relevant AMM Survey (?) ICRA 2000: Grasping and Manipulation Review [Bicchi and Kumar, 2000]

11. Two other possibilities • Problems are too hard • Or • Nobody is interested in funding this work!

12. Significant Accomplishments: Industry Remember those ~9880 ICRA papers? • Fanuc • 20% market share • 1800 employees (1300 in research labs, 10 Ph.Ds) • 10,000 robots • Technology provides the competitive edge • Before • servo motors/amplifiers • Now • collision detection, compliance control, payload inertia/weight identification, force/vision sensing/integration • robots assemble/test robots • beyond human performance Technology transfer does happen! And mobile manipulation!

13. Results we can build on…(a parochial view) • Modeling/controlling humanoids • Dynamic manipulation and locomotion • Cooperative mobile manipulation • Distributed locomotion (and manipulation) systems • Haptics and teleoperation

14. Humanoid dynamics and control • Integration (composition) • Integrated control of reach and posture • Task space versus posture space • Biomechanics for robotics • Realistic models • Minimum principles leading to realistic motions [Khatib]

15. Humanoid dynamics and control • Whole-body multi-contact control • Multiple frictional contacts • Models • Posture • Legs • Locomotion [Khatib]

16. Locomotion and Dexterous Manipulation • Dynamic manipulation and locomotion • Intermittent interaction • Passive dynamics • Reactive control [Rizzi]

17. Significant Accomplishments: Academia • Multiple Mobile Manipulators • Multiple frictional contacts • Maintaining closure [Khatib] [Kumar] [Rus]

18. M3 Modular Mobile Manipulation • Self-organizing, self-assembling, self-repair • Adapt structure • Multiple Functionalities • Can do work [Rus]

19. Teleoperation and Haptics • High-DOF telemanipulators • Locomotion Interfaces [Hollerbach]

20. And yet significant challenges remain! • No successful field deployment of mobile manipulators • Example: Robotic servicing of Hubble (NAS Committee: Brooks, Rock, Kumar) • ETS-VII (JAXA/NASA) • Model-based tele-manipulation • Visual servoing for acquisition of non cooperative targets • No robot (product) capable of physical interactions in unstructured environment • Example: Assistive Robotics

21. Assistive Robotics • Impact • > 5 million wheelchair users* in the U.S. • > 730,000 strokes/year (2/3 disabled five years after stroke), > $50B/year • > 10,000 SCI/year (most < 20 yrs old) • Realistic • Human-in-the-loop • No competing technology • Many other overarching challenges *Inter Agency Working Group on Assistive Technology Mobility Devices

22. Current technology • Artificial limbs: peg legs, hook hand • Crutches, canes, walkers • Wheelchairs • Environmental control systems • Remote control • Many, many customized products

23. Significant Challenges, Problems • 1. New hardware, systems • 2. Modeling/control • 3. Composition, synthesis • 4. Model-based versus data-based

24. Challenge: 10x reduction in effective inertia >20 cm compliant covering pHRI: Safety and Performance [Khatib]

25. Haptic Interfaces and Mobility • Energetic/force interactions between robots and humans • Control simulations or real devices • Personal assist or amplification devices • Rehabilitation or exercise robots • Need haptic interfaces that allow manipulation while walking • Psychological argument for VR • Need to control robots that can reach/grasp/manipulate/lean/kick/push [Hollerbach]

26. Portable Haptic Interfaces • Body-worn systems • Powered exoskeleton • Ground-based system with locomotion interface

27. Representation and Control • Physics of environmental interaction • Distributed interaction • Whole arm/leg/body • Task representation for non-rigid interaction and manipulation • Control and task allocation of multi-function appendages (feet, legs, hands, arms, etc.) • Composition of closed-loop (perception/action) behaviors [Rizzi]

28. Composition of Behaviors: Example • Four behaviors (closed-loop controllers) • Pre-shape (open/close) • Grasp/release • Reach/retract • Go to (move)

29. Composition Pre-shape (close) > Retract

30. Composition Retract > Move

31. Composition Move || Pre-shape (open)

32. Composition Move || Pre-shape (open)

33. Composition Pre-shape (open) > Grasp

34. Composition Grasp > Retract || Move

35. Composition Move

36. Composition Move > Reach > Release

37. Composition

38. Distributed Approaches and Modularity • Distributed Control • Heterogeneous systems with active modules, passive modules, and tools for mobile manipulation • Mobile sub-assemblies and hierarchical control Thanks to Hod Lipson

39. Future Concept for Modular Robotsin Mobile Manipulation Concept: self-assembly with active grippers and rods Concept: mobile sub-assemblies note: mobile manipulation with dynamic kinematic topology for c-space Concept: self-inspection and self-repair with tools

40. Distributed Approaches and Modularity Challenges • Control for systems with dynamic kinematic topology • Under-constraint systems with continuum of solutions • Control for systems with changing c-space • Geometrically-driven posture control • Control for keeping balance and structural integrity • Optimal morphologies for tasks • Uncertainty and Error in Modular Systems • Cooperative approach to error recovery in module and structure alignment, connections, assembly, and repair • Dynamical models with uncertainty

41. Model-based vs. Data Driven • Control/representation • Model based (~15%) • Data driven approaches (~5%) • Dynamic models are getting more complicated and increasingly sensitive to parameters (uncertainty) • Emphasize completely data-driven approaches

42. Discussion • Are there a set of basic research questions that • We can rally around? • Are unique to autonomous mobile manipulation? • Are critical? High-impact? • If so, can we create a new research program? • How do we sell it? • How do we take this to the next step? • Balance • basic research • high-caliber applied research • How do we make robotics a “big science”?

43. Intellectual Basis for New Programin Autonomous Mobile Manipulation • Closed-loop behaviors • Perception-action loops • Vision-based control • Composition of behaviors • Sequential • Parallel, hierarchical • Task description language • Formal semantics • Uncertainty • Understanding and characterizing uncertainty • Data-driven approaches • Teleoperation and haptics • Integration mobility with manipulation Can it be a Tether-esque program?