Shared Control for Dexterous Telemanipulation with Haptic Feedback

1. Shared Control for Dexterous Telemanipulation with Haptic Feedback Weston B. Griffin Dissertation Defense Presentation May 1, 2003

2. Telemanipulation • First systems developed ~ 1940’s • handling radioactive materials • Can provide access todangerous environments • Benefit from natural human abilities operator master slave environment [The E1 developed by Goertz at Argonne National Lab]

3. Telemanipulation • Applications include: • underwater salvage • nuclear waste handling • space station repair • minimally invasive surgery [Intuitive Surgical, Canadian Space Agency, Oceaneering International]

4. position force Telemanipulation Frameworks • computer controlled electro-mechanical systems remote controlled robot <> feeding back information • several different architectures Operator Master System Slave Controller Slave Manipulator extends a person’s sensing and/or manipulation ability to a remote location

5. position force Manipulation • Desire to leverage human manipulation skills • immersive hand/finger based system Master System Slave Manipulator

6. Movie • Remote Control by Andy Shocken • filmed 2002 in our lab • narrated by Mark Cutkosky

7. Master system design • difficult task considering complexity of human hands • active area of research • Enhance slave controller • by sharing control between operator and slave system - shared control Issues in Telemanipulation operator may feel remotely present BUT is not getting normal manipulation cues • Current telemanipulation limitations • force feedback (limited accuracy and fidelity) • limited tactile display

8. Contributions • Development a human-to-robot mapping method • map glove-based hand motions to a planar robot hand • Development and implementation of a shared control framework for dexterous telemanipulation • combining operator commands with a semi-autonomous controller • Investigation of an experimental telemanipulation system • results demonstrate benefits of shared control and need to choose carefully types of feedback to achieve a real improvement

9. Outline • System overview • Human-to-robot mapping • Shared control framework • Experimental investigation Development of dexterous telemanipulation system

10. Improving Telemanipulation • Take advantage of the slave controller and local sensor information for improved dexterity • add “low-level” intelligence Why? • can feedback sensor information by other means • robot can intervene in certain situations (fast response) • human and robot can share control for improved performance

11. force position commands sensor feedback Shared Control bilateraltelemanipulation high level commands & feedback semi-autonomous dexterous manipulation

12. Shared Control combining operator high level and low level commands with a remote controller for improved manipulation

13. Hand tracker CyberGlove CyberGrasp Master System • CyberGlove™ instrumented glove • 22 bend sensors • calibrated for dexterous manipulation [Turner 2001] • CyberGrasp™ fingertip force feedback • lightweight exo-skeleton • uni-directional force feedback • Logitech hand tracker • ultrasonic transducers and sensors • 6 d.o.f. position and orientation [CyberGlove and CyberGrasp are products of Immersion Corporation]

14. Slave System • Custom built robot hand • two fingers, two d.o.f. per finger • low inertia DC motors • cable capstan drive • Robot arm • Adept industrial arm, five d.o.f. • enlarges task workspace • Fingertip sensors • two-axis force sensors • contact location sensors

15. 1000 Hz 200 Hz 1000 Hz 50 Hz 200 Hz 7 Hz 63 Hz 1000 Hz / 200 Hz 1000 Hz System Architecture Master CyberGrasp CyberGlove Indirect Feedback Wrist Tracker GUI QNX Node-to-Node QNX-Node 1 QNX-Node 2 Adept Control Slave Slave Control

16. Outline • System overview • Human-to-robot mapping • Shared control framework • Experimental investigation Development of dexterous telemanipulation system

17. Human-to-Robot Mapping • Robot is non-anthropomorphic, symmetric, and planar • joint-to-joint mapping not possible • very different workspace

18. Human-to-robot Mapping • How do you control a non-anthropomorphic robot hand using a human hand and glove? ? ?

19. Virtual Object Mapping • Interpret human fingertip motions to be imparting motions to a virtual object held between the fingers • Virtual object parameters are mapped to robot • to produce fingertip positions OR motions of a grasped object • Parameters independently modified • to account for kinematic and workspace differences

20. 0 -0.05 -0.1 -0.15 Mapped Pinch Point Position -0.2 -0.25 -0.25 -0.2 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25 Index Mapped Positions Left Finger Boundary Thumb Mapped Positions Right Finger Boundary Virtual Object Mapping • Match natural human manipulation motions to corresponding robot hand motions • good mapping? • operator can intuitively control robot and utilize robots workspace

21. Outline • System overview • Human-to-robot mapping • Shared control framework • Experimental investigation Development of dexterous telemanipulation system

22. Shared Control • Hannford et al. [1991] • force feedback joystick controlling robot arm/gripper • improved task completion time and resulted in lower forces • Michelman and Allen [1994] • sequencing primitives for dexterous hand control • joystick control, no provisions for haptic feedback • Williams et al. [2002] • NASA’s Robonaut project - robot arm and dexterous hand • force feedback joystick for control • reduced task peak forces

23. Shared Control Next step: using shared control in a dexterous telemanipulation system with fingertip force feedback How? • implement a semi-autonomous controller capable of dexterous manipulation • robot has force and tactile sensors and specialized control laws for manipulation

24. Dexterous Manipulation • What does it mean to autonomously manipulate an object? • with sensors robot can detect the object and determine proper fingertip forces for: manipulation

25. Dexterous Manipulation • What does it mean to autonomously manipulate an object? • with sensors robot can detect the object and determine proper fingertip forces for: manipulation

26. Dexterous Manipulation • What does it mean to autonomously manipulate an object? • with sensors robot can detect the object and determine proper fingertip forces for: manipulation grasp force regulation

27. Velocity Grasp Transform Ts + + ZOH Tactile Based Object Tracking - Tactile Sensing + Object ImpedanceController + + Forward Grasp Transform Finger Controller RobotFinger Internal ForceController + - Internal ForceDecomposition Object Manipulation Control • Utilize the Grasp Transform to determine robot fingertip forces [Mason & Salisbury 1985]

28. Object Manipulation Control • Controlling internal force Velocity Grasp Transform Ts + + ZOH Tactile Based Object Tracking - Tactile Sensing + Object ImpedanceController + + Forward Grasp Transform Finger Controller RobotFinger Internal ForceController + - Internal ForceDecomposition

29. Velocity Grasp Transform Ts + + ZOH Tactile Based Object Tracking - Tactile Sensing + Object ImpedanceController + + Forward Grasp Transform Finger Controller RobotFinger Internal ForceController + - Internal ForceDecomposition Object Manipulation Control • Controlling object position

30. Shared Control Telemanipulation • What are the advantages to programming robot for dexterous manipulation? • robot can monitor operator’s object manipulation • if necessary, robot can intervene (take over control of object manipulation) • impedance modification, limit motion, prevent release • robot can warn/inform operator of manipulation status through indirect methods • using other feedback modalities (visual indicators, audio, or augmented haptic feedback)

31. Shared Control Telemanipulation • What are the advantages to letting robot take control over force regulation and/or object manipulation? • operator can focus on behavior of grasped object or tool • master commands are no longer essential to prevent unwanted slip or damaged objects • operator can still override to release or grasp more tightly

32. Shared Control Telemanipulation Shared control implementation issues • as the robot assumes more control • concern the operator’s sense of presence will be reduced • we want to keep the operator “in the loop” • preserve operator’s intent • what type of indirect feedback is most effective? • does sharing control improve performance in an immersive fingertip force feedback system? • To answer these questions we perform a set a controlled experiments

33. Outline • System overview • Human-to-robot mapping • Shared control framework • Experimental investigation Development of dexterous telemanipulation system

34. Previous Experimental Studies • force feedback evaluation • Turner et al. 2000: block stacking and knob turning • force feedback with CyberGrasp not always a benefit • Howe & Kontarinis 1992: fragile peg insertion task • audio buzzer sounded if grasp force excessive • operators were not able to reduce force • shared control evaluation • Hannaford et al. 1991: peg insertion task • operator’s controlled position, shared orientation control • reduction in task completion time and insertion forces

35. Experimental Hypothesis • Addition of a dexterous shared control framework will increase an operator’s ability to handle objects delicately and securely compared to direct telemanipulation

36. Experiment Description • Motivating scenario: recovering an ancient Greek vase on the sea floor “fragile object handling” - user’s asked to carry an object with minimal force but without dropping the object

37. Experimental Task

38. If operator’s desired (commanded) force is too low robot can monitor and warn the operator OR robot can intervene and regulate grasp force to prevent object dropping Experiment Description • To assist operator in fragile object handling taskthe robot computes the minimum grasp force required

39. Shared Controlled Task • Operator maintains manipulation control

40. Shared Controlled Task • Operator maintains manipulation control • Robot and operator share control over internal force • robot monitors excessive force

41. Shared Controlled Task • Operator maintains manipulation control • Robot and operator share control over internal force • robot monitors excessive force • robot can apply minimum internal force required to prevent slip

42. Sharing Control in Fragile Task • Target window with intervention can be wider: desired force can drop below fint,min without adverse effects • In theory, it is possible to always do better without intervention

43. Question that arise... • Does warning the operator of a possible failure help? • Does task performance improve with robot intervention? • If robot intervenes, is it necessary to inform operator? • Is it helpful to feed back information of impending state changes (such as object release)? • With haptic feedback in a force control task, what forces should be fed back?

44. Case Effects • Audio Alarms - when operator’s desired force is too high or too low • Robot Intervention - robot assumes control when operator’s desired force falls below a threshold (safe minimum internal force) • Visual Indicator (fingertip LEDs) - to inform the operator of robot intervention • Force Feedback: actual vs. commanded - during robot intervention, forces to operator’s fingertip are reduced (reduced force feedback)

45. Experiment Cases

46. Case Effects

47. Experimental Procedure • Diverse set of subjects • 11 subjects total • 8 males and 3 females • Two sessions • first - calibration and training • second - four trials for each case • Case order randomized • reduce possible learning and fatigue effects

48. Evaluating Performance • Objective data analysis • measured internal force applied to object • fragile object task - lower is better • task failures (number of drops) • task completion time • Subjective data analysis • operator’s expressed preference • operator’s perceived difficulty

49. Measured Internal ForceDesired Internal Force Minimum Internal Force Case 1 4 Force [N] 2 0 0 5 10 15 20 25 Time [sec] Measured Internal ForceDesired Internal Force 101% of Minimum Internal Force Case 2 Excessive Force Warning (Low Tone) Object Slip Warning (High Tone) 4 Force [N] 2 0 0 5 10 15 20 25 Measured Internal ForceDesired Internal Force 110% of Minimum Internal Force Excessive Force Warning (Low Tone)Object Release Warning (High Tone) Robot Intervention C Case 6 A 4 D E B Force [N] 2 0 0 5 10 15 20 25 Time [sec] Typical Subject Data

50. 3.0 Average of allsubjects for each case 2.8 2.6 2.4 Internal Force [N] 2.2 2.0 1.8 average minimum internal force to prevent object slippage 1.6 1 2 3 4 5 6 7 Case Number Data Analysis • Measured internal force applied to the object • averages of each subject for each case (trial failures excluded) • Boxplot • medians and quartiles • observe trends • Is there a significant effect?