Reinhold Behringer Johannes Christian Andreas Holzinger Steve Wilkinson

1. Some Usability Issues of Augmented and Mixed Reality for e-Health Applications in the Medical Domain Reinhold Behringer Johannes Christian Andreas Holzinger Steve Wilkinson

2. Purpose of this Paper • Highlight AR and MR applications in the Medical Domain. • Point to usability issues. • Not a complete survey. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

3. Augmented Reality (AR) • Principle: • Computer generates output, which is fused with the human perception of the environment. • Appears to emanate from the environment itself, to be a part of it – registration with real environment. • Visual: • 3D computer graphics, employing Virtual Reality (VR) concepts. • Acoustic: • Spatial audio. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

4. Mixed Reality (MR) • Real environment is not necessarily the basic framework. • Virtual Environment (VE) is the main framework. • Objects in MR can be computer-generated or real. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

5. AR and MR in Medical Domain • AR: • Data visualisation directly on patient. • Guidance to physician for medical procedures. • MR: • Simulation and training. • Patient itself can be simulated. • Benefits: improved situational awareness Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

6. Display Types for AR • AR fusion of visual sense: • Head-worn displays: • Ideal merging of computer output with visual impression of surroundings. • Hand-held displays: • Either acting as “frame into the real world”, or • Showing representation of real world by video / images, onto which information is overlaid. • Projective displays: • Project information directly onto object (patient). • 3D is only correct for one single user at a time. • Requires consideration of projection surface. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

7. Tracking of User • User’s viewing position and orientation needs to be tracked for correct display rendition. • Technologies: • Active illumination outside of human-visible spectrum (e.g. IR). • Computer vision approaches, capturing either: • Environment as seen from user (head-worn camera), or • User himself, seen from cameras in the environment. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

8. Examples of Applications in Medical Domain • Data projection onto patients. • Support of surgery. • Simulation tools for teaching and practising. • Therapy and rehabilitation. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

9. Visualisation of Ultrasound and CT • UNC, A. State (1992). • Live image stream of ultrasound scans onto body: pre-natal care, obstetric examination. • Initially 2D scans, later volumetric scans. • Technique also used for needle biopsies and minimally invasive surgery. • Stockmans (2005). • Correction of bone disformities. • Comparison of before-after procedure. UNC AR demonstration Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

10. Simulation • Sielhorst et al., TU Munich (2004) • Birth simulator, using MR techniques. • Real object: model of torso. • Graphical simulation: baby. • Birth pliers as haptic interface for interaction. • Nestler et al., TU Munich (2007) • Virtual patients for large-scale disaster training. • Table-top display, individual patients with different injuries. • Wilkinson (2005) • Using purely VR for simulation of surgical procedure on hand. • Goal: to educate patients and reduce fear. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

11. Aiding Surgery • Not yet practical in clinical applications, but promising improvements of spatial orientation, allowing more radical operative therapy. • MEDARPA project: • AR and VR supporting minimally invasive surgery, to aid the navigation of the surgeon (bronchoscopy and brachytherapy). • Future use with surgical robot. • Reitinger, TU Graz (2005) • Virtual liver surgery planning system, aiding in providing more precise measurements for tumor treatment. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

12. Therapy, Rehabilitation • Using VR to cure phobias. • Learning in controlled graphical environment how to deal with phobia-causing situations. • Rehabilitation of motoric skills: • AR system evokes motor images, • Guides motoric practise • Healthy living: • “Persuasive Mirror” shows future projection of self, under influence of varying live style. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

13. Edutainment • Teaching anatomy: • “electronic book” for interactive display of 3D anatomic perspective of humans. • Organs can be viewed from different angles. • Simulation: • To reduce fear of the unknown. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

14. Registration and Tracking • Requirements: • View of patient’s body needs to be unobstructed for the physician. • Tracking needs to take into account deformable tissue of human body. • Calibration effort needs to be small, so as not to distract from medical procedure. • Tracking needs to be resilient to occlusion. • Markers for visual tracking can be attached to human body. • Magnetic tracking can provide seamless tracking of physician. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

15. Displays • Head-attached: • Allows hands-free activity. • Low resolution and viewing angle. • Example: Variscope (Birkfellner, TU Vienna) • Hand-held: • Is currently available at reasonable cost and specifications. • May interfere with actual procedures. • Tablet display on boom: • Allows hands-free operation, while at the same time being cost effective. • Can be semi-transparent or video-see-through. • Both display and physician need to be tracked for correct visual view. • Spatial. Variscope MEDARPA display Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

16. Interaction • Traditional interfaces (mouse, keyboard) are not suitable, as they distract from the task. • Preferable: • Automatic ubiquitous interface, acting in the background. • E.g. speech recognition, gesture recognition. • Possibly integrated into medical instrument. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

17. User-Centred Development • Potential benefits of AR/MR are obvious. • But for studying usability, practical workflows need to be studied. • AR application developers need to understand the workflows of medical professionals. • Usability Engineering Methods (UEM) are becoming more important, but are still not applied very often: • Software engineers work on the implementation, usability experts work on design. • Rarely do they collaborate. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

18. User-Centred Development • In most cases: • Usability of AR environments is evaluated. • E.g. Hix: • Criteria for assessing VE design. • Usability of standard user interfaces for VE. • Effectiveness of different interactive devices. • Needed: • User-Centred Development. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

19. Presence and Errors in VE • Specific to VEs: • Sense of presence (immersion). • Assessed through questionnaires, focusing on a person’s sense of presence. • Presence is not associated with task performance (Slater). • But measures of presence have concentrated on user perception of VE technology. • Errors in VEs can impact significantly the perception of presence Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

20. Immersion • Effects of being immersed: • Investigated in a variety of VE system configurations: • Collaborative VEs, viewpoint of self, perception of others’ presence. • Fully immersive VEs have been well investigated, focusing on head-worn displays. • Missing: • Techniques for measuring presence. • Questionnaire-based assessment may be biased. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

21. Concerns about AR • Problematic: • Information overload can be accompanied by sensory overload. • Overwhelming experience in AR environment. • Possible lack of acceptance of this technology. • Social effects (beliefs, attitudes, feelings). Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

22. Suggestions for Further Study • Which are the most appropriate interaction metaphors in AR for medical domain? • Which influences do AR applications have on performance of the end users in the medical domain? • What are the effects of adaptation that people might have to make cognitively to believe in and cope within an AR environment? • How will multimodal interaction through a number of input and output channels enhance or detract from the reality/Virtuality experience? Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

23. Specific to AR • AR must deal with • physical interactions, • social interaction, • cognitive interactions. • Is difficult to isolate variables. • Less performance constraints. • Lower predictability of behaviors. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

24. Successful AR Application • All involved research domains have to be considered. • User-centered design focus is important. • Need to be: • Accessible. • Usable for everyday end users. • Follow notions of pervasive and ubiquitous computing. • Implement basic ideas of social software. • Designed for users without deep IT knowledge. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

25. Summary • AR and MR have excellent potential in medical applications. • Usability issues need to be addressed in application design: • Not only study usability in experience of VE environments. • Need to apply usability engineering methods in design. Leeds Metropolitan University Innovation North – Faculty Of Information And Technology

26. Acknowledgment • Johannes Christian • Andreas Holzinger • Steve Wilkinson Leeds Metropolitan University Innovation North – Faculty Of Information And Technology