Modélisation de l’interaction avec objets déformables en temps-réel pour des simulateurs médicaux

1. Modélisation de l’interaction avec objets déformables en temps-réel pour des simulateurs médicaux Diego d’Aulignac GRAVIR/INRIA Rhone-Alpes France

2. Medical Simulators • Motivations • danger to patients • cost • certification • Objectives • Geometric Models • Physical Models • deformation • interaction

3. Problems • Simulation MUST be real-time! • deformation • resolution • Simulation MUST be realistic! • model • identification of parameters • Simulation MUST be interactive! • collision detection • haptic interaction

4. Plan • Deformation Models • Mass-Spring vs. FEM • Real-time Resolution Techniques • Static • Dynamic • Echographic Simulator • parameter identification • Liver Model • interactive deformation

5. Deformable Object • Geometry • Elements • Springs [TW90] • Tetrahedra FEM [OH99] • Comparison • Realism • Speed

6. Geometrical Model • 56 surface points • 108 triangles • 57 total points • 120 tetrahedra • 230 edges

7. Mass-Spring Model Initial length Deformed length

8. Finite Element Method (FEM) Deformed configuration Deformation tensor: Initial configuration x a Green’s strain displacements Small strain Cauchy Strain:

9. Strain-Stress Deformation Energy Lamé coefficients force per unit area

10. Mass-Spring Model • Springs are placed along the edges (230) • Not very realistic: modeling a volume with springs! • The force of each spring relatively cheap to evaluate • globally fast

11. Finite Element Method (FEM) • 120 tetrahedra using Green’s strain tensor • Continuum is modeled with volumetric element. • Dilatation may be controlled • Approximately four times slower than mass-spring network

12. Deformable Models (conclusions) • Mass-Spring • One dimentional elements • Unrealistic to model volume • Tetrahedral FEM • Good realism for 3D continuum • Control of dilatation • Approximately 4 times slower to evaluate forces

13. Contributions • Quantitative and qualitative comparison of mass-springs and tetrahedral elements • Interactive non-linear static resolution • Formal analysis of the real-time stability of integration methods • based on parameters • Identification of the parameters of a model from experimental data • Relevant medical applications

14. Plan • Deformation Models • Mass-Spring vs. FEM • Real-time Resolution Techniques • Static • Dynamic • Echographic Simulator • parameter identification • Liver Model • interactive deformation

15. Real-time Resolution • Static Resolution • linear resolution [Cotin97] • small displacements • Our approach: non-linear resolution • large displacements • Dynamic resolution • explicit [Picinbono01] • implicit [BW98]

16. Linear Static Resolution • Linear case: • Pre-inversion (if enough space) • No large strain • No rotation • No material non-linearity Principle of virtual work: internal and external forces are balanced

17. Nonlinear Static Resolution • Non-linear case: • Stiffness matrix changes with displacement: • geometric • material

18. Newton Iteration • Full Newton-Rapson method: • Reevaluation of Jacobian • Faster convergence • Modified Newton-Rapson method: • Constant Jacobian • Slower Convergence

19. Dynamic Analysis 2nd order non-linear differential equation Convert to 1st order system

20. Explicit Integration Runge-Kutta method with s stages s Order of consistency (accuracy) vs. stages precision

21. Explicit Integration Stability linearizing Im Timestep is limited by the the physical parameters! Re

22. If you know your history,then you would know where you are coming from. Bob Marley Implicit Integation Over-damped case B-stable implicit euler: linearisation Semi-implicit euler Stable for linear case (A-stable) any timestep any physical parameters

23. Resolution (conclusions) • Static analysis • non-linear resolution for large displacements • Dynamic • explicit • strict stability criteria • implicit • no limit on timestep, but resolution of non-linear system

24. Contributions • Quantitative and qualitative comparison of mass-springs and tetrahedral elements • Interactive non-linear static resolution • Formal analysis of the real-time stability of integration methods • based on parameters • Identification of the parameters of a model from experimental data • Relevant medical applications

25. Plan • Deformation Models • Mass-Spring vs. FEM • Real-time Resolution Techniques • Static • Dynamic • Echographic Simulator • parameter identification • Liver Model • interactive deformation

26. Thigh Echography

27. Echographic Simulator • Data Acquisition • Model of the thigh • Mass-Spring • Neural • Interaction • collision • haptics • Generation of echographic image

28. Data Acquisition (at LIRMM, Montpellier) 64 sample points are marked on the thigh. For each, the forces for some given penetrations are measured Two different probes (a) Indentor shaped probe for punctual force-penetration data (b) Probe with surface equal to that of a typical echographic probe 1- The end effector advances in small steps (2mm) in the direction normal to the surface of the thigh. 2- The force depending on the penetration distance is measured

29. [d’Aulignac et al. MICCAI 99] Force Force displacement displacement Data Acquisition: Experimental Results • The two probes do not offer the same resistance • difference in surface area • Different curves for different points • different depth of soft tissue • Highly non-linear behaviour Indentor probe Surface probe

30. Echographic Simulator • Data Acquisition • Model of the thigh • Mass-Spring • Neural • Interaction • collision • haptics • Generation of echographic image

31. Dynamic Model of the thigh Incompressibility of the tissue Elasticity of the epidermis • Why mass-spring model? • computationally efficient • interior NOT discretized into tetrahedra

32. Identification of the Parameters of aDynamic Model OptimizationAlgorithm New parameters (elasticity, plasticity, collision stiffness ...) Error - Behaviour Resolution Model Desired behaviour Measurements For each sample point, 10-12 deformation/force values with each probe => Total of ~1200 measurements.

33. (in collaboration with UC Berkeley) [d’Aulignac et al., IROS 99] Distribution of Nonzero Error Values Parameter Estimation Least-squares minimisation: 1. find (a,b) for each non-linear spring 2. find (a,b) for each non-linear spring, and (a) for all linear springs => Avoid local minima • Error of the model with respect to the experimental data => Overall error less than 5% Error (N)

34. Explicit integration Euler stability too small timesteps no real-time ...or large mass slow movement no gravity Implicit integration Semi-Implicit Euler constant Jacobian 100 steps per second h=1/100 (i.e. real time) Dynamic Analysis

35. Dynamic Resolution 100 Hz using semi-implicit integration

36. Neural Networks Displacement of particles: u • Static Analysis • Multi-layer perceptron is a general approximizer • Network is trained directly on experimental data • back-propagation Forces acting on particles: f 64 inputs and outputs

37. Neural Networks Displacement (mm) Force (N) Neural Model Experimental data

38. Mass-Spring vs. Neural Model • Mass-spring • topology chosen • based on measurements • dynamic resolution • semi-implicit (100 Hz) • Neural model • no assuption on topology • static resolution • very fast • no change of topology

39. Echographic Simulator • Data Acquisition • Model of the thigh • Mass-Spring • Neural • Interaction • collision • haptics • Generation of echographic image

40. Interaction • Collision Detection • Collision Response • Force Feedback

41. Collision Detection • Finds polygons in the OpenGL viewing frustrum • Detects collision between simple rigid body and any other object quickly

42. Collision Response Penalty forces [Hunt and Crossley 1975] • Inter-penetration distance must be computed • Generates large forces (bad for haptics)

43. Haptics • Haptic devices require high update frequency • typically around 1kHz • ….which the simulation normally can’t meet • 100 Hz (dynamic model)

44. Haptic Interaction • Local approximation of the contact • simple local model running in a separatethread • fast collision detection • fast force computation Haptic loop (1kHz): collision detection and response with local model [Balaniuk 99] Local model update position Simulation Loop (100Hz): deformation global collision detection and response

45. [d’Aulignac et al. , ICRA, 2000] Haptic Feedback With local model force time Without local model

46. Echographic Simulator • Data Acquisition • Model of the thigh • Mass-Spring • Neural • Interaction • collision • haptics • Generation of echographic image

47. Echographic Image Generation [Vieira01](in collaboration with TIMC-IMAG, France) • 64 images aquired • on each sample point • Voxel Map • 120 Mb • Interpolation • fill in the blanks • Provide image • any rotation • any position

48. Echographic Image Deformation • Problem • structures deform differently • vein • bone, etc. • segmentation • Linear deformation • Possible extension: precalculated deformation maps [Troccaz et al, 2000]

49. A first Prototype

50. Echographic Simulator (conclusions) • Data Acquisition • Model of the thigh • Mass-Spring • Neural • Interaction • local model • Generation of echographic image • linear deformation