Seeing and Sensing: displays, sensors, and systems for medical interaction

1. Seeing and Sensing: displays, sensors, and systems for medical interaction Michael Halle, PhD Surgical Planning LabDepartment of Radiology, BWH mhalle@bwh.harvard.edu

2. Acknowledgements S. Pieper M. Shenton F. Talos A. Tannenbaum C. Tempany S. Warfield W. Wells C.F. Westin and the rest of the SPL… N. Aucoin P. McL Black K. ChinzeiE. Grimson S. Haker K. Hynynen F.A Jolesz R. Kikinis L. O’Donnell NCRR, NCI, NLM, NSF, DOD et al. http://www.spl.harvard.edu

3. Computer hardware and software is becoming more advanced, but getting dynamic information in, and meaningful information out, is still hard. Observation:

4. The I/O problem

5. Result • poor integration of important data + information may not completely and accurately get to physician = doctor cannot make most informed decisions & innovations aren’t easy

6. What we’ll do about it • new display technologies to understand complex data • software systems that simplify data exchange between different systems • collaborative software tools

7. Who am I? • MIT Computer Science (BS) • World’s most advanced 3D displays at MIT Media Lab (MS, PhD) • Systems and graphics design • Long collaboration with SPL • Started full-time 1995

8. Holography and 3D

9. Graphics first • For the last ten years, better graphics has been more important than 3D displays • Now, we can start to build systems that integrate the best of 2D and 3D displays

10. Several candidate technologies • Volumetric displays(eg, Actuality Systems) • Space-filling • Collaborative • No opacity Provided by Nakajima, Atsumi et al.

11. Female pelvis on 3D display (Hoyte)

12. Stereo glasses / head-mounted display • Can be cheap • High quality imagery • Can’t register with real world easily • Only one user

13. Lenticular and microlens displays • Multiple viewers & walkaround • High image quality • Can be difficult to produce imagery Lenticular Microlens

14. Display approach • None of these displays are perfect for all tasks • Computer and monitor still central • Use each, appropriately • Maintain mental context when switching

15. Maintaining context Provided by Leventon et al.

16. Input devices: data fusion Provided by Leventon et al.

17. Input devices • Good displays need good data • Input means interfacing • Interfacing usually means translation • Analogy: language and society • Power of common language

18. Communication community Central Switchboard translator translator 2D vis. probe Effective path

19. Communication community Central Switchboard translator translator 3D vis. probe 2D vis. translator other contributors

20. Advantages • New “contributors” are simple • Experimentation is easier • More robust • Easy to monitor system • Richness of interaction & computation

21. Extension: collaborative interface • Interface communication • Shared data sources • “Shared mental context” • (Early version a decade ago!)

22. Status • Communication mechanism prototype • Proof-of-concept device interface • Short term: integrate with Slicer • 3-4 months: prototype collaborative interface • 1-2 years: 3D display hardware on site • Other collaborative interfaces

23. Conclusion • For each tool, its task • Technology with purpose • Community communication • The future, piece by piece

24. Contact information Michael HalleSurgical Planning LabDept. of RadiologyBWHmhalle@bwh.harvard.edu

26. Intraoperative Changes Craniotomy Initial Provided by Warfield et al.

27. Visualization • Data Acquisition • Data Analysis • Rendering too little too much just right

28. Core technologies • Data acquisition & preprocessing • Segmentation • Registration • Analysis • Model building • Visualization • Interaction not a linear pipeline!

29. Data acquisition example: diffusion tensor imaging Provided by Westin, Meier, et al.

30. Diffusion Tensor MRI Tractography Provided by Westin, Mamata, et al.

31. Preprocess: feature enhancement Example: 3D adaptive filtering of vessels Original Filtered Courtesy CF. Westin

32. Segmentation • cannot be avoided • based on many criteria • must cope with imperfect signal

33. EM-MRF Segmentation EM EM-MRF [Kapur98] T. Kapur, W.E.L. Grimson, W.M. Wells III, R. Kikinis. Enhanced Spatial Prior for segmentation of Magnetic Resonance Imagery, MICCAI98, Cambridge, MA, October 1998. [Kapur99] Tina Kapur. Model based three dimensional Medical Image Segmentation Ph.D. Thesis, MIT EECS Dept, 1999.

34. Registration • Multiple input channels • Different timepoints • Different patients • Patient to atlas

35. ATM Classification Combine statistical classification and registration of a digital anatomical atlas Brain atlas Registration Template Distance Transforms prototypes Statistical Classification Segmented images Grey value images

36. Atlas Mapping atlas knowledge into patient data sets Provided by Kaus, Nabavi, Warfield

37. Cartilage Thickness Mapping Courtesy Warfield, Winalski

38. Mapping of all information: Brain Morphology Vessel Morphology Brain Function (GM, WM) Pathology External Information Not automatic! Integrated Visualization

39. Interaction • Bring the results to the physician • People are interface too • Changing the look of medicine

40. Intra-operative MRI • Precise targeting of • lesion margins • surrounding anatomical structures • image updates as needed - (“brain shift”) • accurate location of residual tumor • Immediate detection of complications (hemorrhage, swelling, ischemia)

41. Volumetric FE Deformation Provided by Warfield, Ferrant et al.

42. 3D-Navigation Provided by F. Talos

43. Robot & Prostate Focused Ultrasound (FUS) Novel Interventional Activities

44. FUS Setup Provided by Hynynen et al.

45. 3D Slicer • Surgical simulation and navigation software • Displays multi-modality images in 2D and 3D