ENME: An ENriched MEdia application utilizing context for session mobility in a telecom system

1. ENME: An ENriched MEdia application utilizing context for session mobility in a telecom system Lill Kristiansen, Prof. Dr. ScientDept. of Telematics, NTNU, Norway lillk@item.ntnu.nowww.item.ntnu.no/~lillk Mainly based on Master Thesis of Egil C. Østhus specialication in ’Networked Services’ in Telematics Co-supervised by Per-O. Osland, Telenor R&I Health care scenarios, some former work and supervision by L.K.

2. Outline • 2 scenarios (from business and health care) • Background work and theories • Enabling technologies (SIP) (short) • Requirements for the application ENME • Overview of implementation • Discission in a health care perspective (CSCW) • interrupts, choosing the callee, one or many terminals? • Health care specific vs general applications (like IM)

3. Business trip A mobile worker about to enter the airport express train A caller phones him up with MMoIP He answer with PDA/phone Health care Use of MDA, pat. terminal, IM+loc.+ phones Nurse looks up available doc. and phones him up with MMoIP Doc. answers on phone 2 Scenarios caller + patient caller callee callee

4. Business trip Traveller moves to booth with bigger screen, multimedia is added by the ENME service Health care Doc. moves to booth with bigger screen, multimedia is added by the ENME service Some short range ’telemedicine consultation’ takes place 2 Scenarios same callee same callee (different picture!)

5. Comments on scenarios • Telenor had focus on the general case • mobile business traveller as example of ’generic mobile worker’ • wants to use standard SIP, support roaming etc • Lill has focus on a specific case • Bed ward at hospital • nurse-nurse, nurse-doc. , doc.-doc. • Still using SIP standard is relevant, but add CSCW aspects • Student Egil: Needed something working in 5 mo. time • context modeling, SIP • some focus on user aspects (EU-project with Tandberg) for a general office setting with local mobility and virtual teams • a demo using simple lab setting for location tracking • no focus on health care • Not suitable for real testing at bed ward

6. Examples of possible terminals (devices / MDAs) in the hospital setting e-bookfrom Know-mobile Imaginary’planned 2-piece’From Ericsson Partlyfrom RiT 2000

7. Why is my hand tied up with the mobile? use earplug or loudspeakingphone? caller callee moves to utilize video Scen. PocMap vs Scen VI phone + EPJ how much integration? (Non-integrated phone + IT exists today) , seems to be ’fixed’ or ’available’ caller callee less focus on login / read / write in EPR, but ’shared applic.’ may be used

8. Outline • 2 scenarios (from business and health care) • Background work and theories • Terminologies • Mediated communication theories • Media richness, Social presence • User studies and scenarios from health care

9. Terminologies (1/2) • Conversational service is used by 3GPP (and ETSI) to describe non-same place real time communication where the timing requirements support fully natural conversation between two humans.(...) • Real time communication is by some regarded as synonymous to synchronous communication [1] . However, it may also be used somewhat differently for communication with strict timing requirements (such as lip synchronization, detailed jitter requirements etc.), and this is the way it is used in telecom. • Synchronous communication is communication taking place at the same time (as opposed to traditional mail, voicemail etc). • Face-to-face meeting (F2F) is communication in same time and same place. Here the ‘sameness’ of the place is considered to be ‘close’. F2F is normally considered to be ‘without technology’. • [1] It seems that e.g. Bardram in [5] use the term ‘real time’ in this sense without specifying any strict timing requirements.

10. Terminologies (2/2) • VoIP • Voice over IP (e.g. using SIP) • ‘IP-telephony’ / ‘broadband telephony’ • VVoIP • Voice and Video over IP (e.g. using SIP) • ‘Videoconferencing’ over IP • MMoIP • MultiMedia over IP (e.g. using SIP) • ‘Real time conversation and collaboration service’ • Will allow both real time voice and video as well as ‘shared application’ (e.g. sharing a picture, a part of the EPR, a look up in a medical source etc)

11. Mediated communication theory (1/2) • According to the social presencemodel of communication, media can be distinguished according to the degree of interpersonal contact • Note: Does not separate VoIP and VVoIP well (both are ‘medium’) • Does not address the view of the patient ( =/= caller A =/= callee B)

12. Mediated communication theory (2/2) • Media richnessis defined in terms of how well a medium can communicate equivocal or ambiguous information • (e.g. video of the face of the speaker, body language. ‘grounding’ etc) • Face-to-face and video telephony is ‘rich’, and then comes voice telephony, other synchronous media and then asynchronous text. • Hence voice telephony (VoIP) and video telephony (VVoIP) is differentiated regarding media richness. • Robert and Dennis (2005) argues that non same time media will allow the receiver to carefully consider the arguments and that this makes asynchronous non same time communication well suited in many types of arguments and discussions. They refer to this as the ‘paradox of richness’. • In our setting this is of interest since MMoIP allows for flexibility between real time voice and live video (VVoIP) on the one hand and ‘not-so-same-time’ IM, as well as pre-made memos, books and figures on the other hand. • None of these 2 theories takes any broader social view on situations, work conditions etc. Mostly assumed A and B are communicating, again not focusing on the view of patient/C (’ 3rd party’)

13. User studies (1/3) • We may notice that in most cases actual user studies in mediated communication assume that the media is not changed dynamically during the communication session. • May be 1-1 or n-n or 1-n, but ’addressing’ not much discussed in these two theories (some though in ’social presence theory’) • Studies using conversational services are mostly using the phone system ‘as is’, including AwarePhone from Bardarm and Hansen, using a simple ‘click to call’ mechanism • Studies from Computer Science in CSCW more often builds ‘computational services’ (not supporting strict real time and phone services) • Also many user studies do not deal with real time conversational services, but rather with distributed information or computation services Note [1].Note [1]: In Coiera (2000) the terms ’conversation’ and ’computing’ are used.

14. User studies (2/3) • A study from Eye-to-Eye EU-project (Følstad et al.) allowed for ‘office workers’ to chose between • ’avatar phone’, email, multimedia telephony, F2F meeting • i.e. mostly 1-1- communication (except email) • Result: Users seemed quite conscious on the media choice depending on the task at hand. • This at least tells us that ‘the more media the better’ is not always true, and that full real time multimedia is not always the answer. • This fits well with the other studies in mediated communication (such as ‘paradox of richness’ )

15. User studies (3/3) • The Knowmobile research project gave PDAs, laptops and GSM-phones to Norwegian resident physicians and medical students carrying out work in hospitals. • Each user had both handheld devices and PCs. • In Gallis,Kasbo and Herstad (2001) they write: • ‘The multi device paradigm leads to problems connected to the use, design, harmonization, (...) of various devices’.

16. Health care scenarios (1/ ) • Scenario II (Illustrating interrupt/context shift) based on real user studies • Dr. Davis (...) has just arrived at the ward. Almost immediately she is called [1] upon by nurse Neil (using the MEPC[2]) who asks about patient Palmer’s Medication (…). After checking the status of the patient, Dr. Davis is about to enter the medication dose, but then she is called [3] to patient Adams who has had a ventricular tachycardia. She has to go there immediately, leaving the medication of patient Palmer unfinished.(…) (from Dahl, Sørby, Nytrø) • [1] Not by a phone call, but a ‘standard request’ containing name of patient, name of medicine,+ + • [2] MPEC: Mobile Electronic Patient Chart • [3] ‘Standard request’ in paper i.e. a health specific request containing e.g. patient name, medication name ++ . NOT a ‘phone call’/ MMoIP using SIP technology Why not?

17. Health care scenarios (2/ ) • Scen. III and IV (from Favela ++) based on user studies • Scen. III can be read in paper • Scen. IV [Continuation of Scenario III] • While Dr. Garcia is analyzing the patient’s medical condition, he notices on the map that a resident physician is nearby and calls[1] her up to discuss with her this clinical case”. [1]It is not clear from their description whether the call is a phone call or a kind of ’message call’ delivery • This distinction would be interesting from an interrupt point of view. • If phone call: an example of using the phone system ‘as is’

18. Health care scenarios (3/ ) • Cont. of Scen II • While Dr. Davis is working on patient Adams, the alarm goes as patient Taylor gets cardiac arrest. Since Dr. Davis is not available, Dr. Osborn from another ward gets a message on his MEPC . (…). (from Dahl, Sørby, Nytrø, 2004) • Here we note that the system seems to ‘reroute’ the call/alarm initially directed to Dr.Davis (or to role ‘this patient’s doctor’?). I.e. the caller does not determine the callee by person or by static role (rather a dynamic role?) • This has similarities to the scenarios described by one of my students in 2003, see next page

19. Health care scenarios (4/ ) • Scenario V: One nurse needs help with patient and initiates a ‘message call’ by pushing a button named ‘help-with-patient’ on her PDA. The nurse is not specifying any receiver (callee). The effect of this ‘call’ shall be a message delivered to one ‘suitable’ other person within the group, preferable one that is ‘the least occupied’, and ‘suitably close’. In case of ‘no answer’, then next person will be tried. • based on a telematics student Forthun (2004), without any detailed user studies) • Scen. V (from 2003) has some similarities with existing, new PDA based patient signal call system at St. Olav’s, but that system uses no location (and no ‘busy’) only predefined groups of nurses ( i.e. no dynamic context parameters)

20. Health care scenarios (5/ ) • With AwarePhone (by Bardam and Hansen) • Doctors and some other key personnel at a surgical department are equipped with PDAs and location tracking. Other users may wear a ’bug’ for location tracking. • Both doctors and nurses may see and utilize the following context on a wall mounted screen (and on the PDAs with AwarePhone implemented) • presence status (manually set), location (automatically detected) and calendar info (automatically fetched) • Head nurse Ann having a PDA wants to contact Dr. Ben. She looks up his status on AwarePhone. She can then determine to call him (ordinary voice call), or send his a voice mail or a text message. The voice call will go via the public GSM system via ’click to call’, i.e. using the phone system ‘as is’) • Note: Compared to the previous slide the human decides the caller (and the service), no ’choice’ or rerouting by the system.

21. Definition of context • The definition proposed by Dey [2]: • “Context is any information that can be used to characterize the situation of an entity. An entity is a person, or object that is considered relevant to the interaction between a user and an application, including the user and application themselves.”

22. Hierarchical context model Note: Some parts of context are ’objective, technically determined’,while other aspects are socially determined (not easily modeled) Some are health specific: ’medication of patient Palmer with ZZZ’ etc

23. Outline • 2 scenarios (from business and health care) • Background work and theories • Enabling technologies • SIP from IETF

24. SIP: Session Initiation Protocol (short!) • From IETF (Internet Engineering Task Force) • Also used by traditional telecom standardization bodies • 3GPP (with ETSI for UMTS) for use in IMS • IP Multimedia Subsystem in UMTS • 3GPP2 the US variant • Baseline SIP handles call set up • SIP Refer handles ’call transfer’ (used by us) • SIP allows for extensions (used by us) • SIP for presence • SIMPLE (not used here, but relevant for Ubiquitous Comp.) • Relevant for IM/presence systems

25. Outline • 2 scenarios (from business and health care) • Background work and theories • Enabling technologies (SIP, SD, loc. etc.) (short) • Requirements for the application ENME

26. Requirements of ENME (high level) • The ENME service shall be a value added service on top of an all-to-all telephony service. It shall work with a major standard MMoIP protocol. • ENME shall detect when a more suitable context occurs (i.e., fitting the proposed media types better). ENME shall then suggest an upgrade of the media types by moving the session to the new terminal(s). • The ENME service shall support mobility. • ENME shall work in a business case, i.e. it shall be possible to make a sensible charging of the value added service, as well as of the network resources used to transport the media types. • Substantial parts of ENME shall be demonstrated in June 2005 after 5 mo. work. • ENME shall use a generic context model.

27. Scenario VI (using ENME) • Scanario VI Nurse Carla is visiting patient Donna in her room. Together they are discussing the information about potential harmful (but rare) side effects of some medication. Due to this information patient Donna shows Carla his swollen leg. Carla finds this leg so bad that she decides to call the physician. • She checks the availability of the doctors, and finds Dr. Eve to be available. Carla calls her using the patient terminal, and voice and video are chosen as the media types. Dr. Eve is available, but not in her office, so the call arrives at her PDA, and is started with voice only. Due to the oral information Dr. Eve realizes that she needs a closer look at the patient. • Dr. Eve enters a multimedia booth in the corridor. ENME adds video. After a short voice and video session Eve determines that the situation is not acute. She notices that she shall check the situation during her planned visit to the patient later on the day. She then continues with her other activities. • Scen. VI assumes video camera on patient terminal, as well as some location infrastructure and context awareness

28. Outline • 2 scenarios (from business and health care) • Background work and theories • Enabling technologies (SIP, SD, loc. etc.) (short) • High Level Requirements for the application ENME • Overview of implementation • Architecture and animation

29. SIP entity Context/ENME Architecture overview

30. Voice Voice Video Design /animation !

31. Outline • 2 scenarios (from business and health care) • Background work and theories • Enabling technologies (SIP, SD, loc. etc.) (short) • High Level Requirements for the application ENME • Overview of implementation • Discission in a health care perspective (CSCW) • interrupts, choosing the callee, priority? • Health care domain specific applications vs general applications (like IM)

32. CSCW issues (1: interrupts) • Interrupts in our context may be of at least two types: • Automatic changes from a ‘smart’ system might cause interrupts (and so may proposals to perform such changes as well). • These are not handled, timers may help • We assume the ENME subscriber is actively searching for a better place (not arbitrarily ’walking around’), • Issues of entering own office for short time and then leaving again must be looked into, ‘active registration’ (but maybe forgetting?) vs ‘automatic detection’ . • Another type of interrupts is caused by the calls and messages themselves, i.e. by VoIP / MMoIP as such. • Phone calls, F2F meetings and IM all have interruptive aspects [6], [58] and [50]. • IM maybe less interruptive in a hospital setting than in the general case.

33. By combining ENME with location and awareness (as in AwarePhone or advanced IM systems with ‘roster’) the problem of manually finding a suitable callee should be decreased But it need human interaction (‘thinking’, ‘load’) Manual status update (as in AwarePhone) may be suitable for doctors and ‘office workers’, Maybe less so for nurses? CSCW issues (2: choosing callee manually) FromAwarePhone (Bardram&Hansen) Intended for ’office workers’ fromwww.cisco.com/univercd/illus/1/48/170948.jpg

34. CSCW issues (2: choosing callee / automatic) • Or having the system to ‘decide everything’ • While Dr. Davis is working on patient Adams, the alarm goes as patient Taylor gets cardiac arrest. Since Dr. Davis is not available, Dr. Osborn from another ward gets a message on his MEPC [i.e. system automatically reroutes the call/request/message] (Dahl et al., 2004) • Requires to do more proprietary solutions, may be more tailored to the actual needs • But may be a ‘lock in’, not updated as new IM technology for the general market evolves

35. CSCW issues (3: priority) • Today different phones and pagers (role based) are used to handle priority to some extent • Priority calls (prio 1,2 3?) is not in standard SIP neither • But ongoing discussion in EMTEL group • Is it needed (see next bullet) • Alternative to a system with prio integrated is the following: • 3 ‘phone numbers’ may be used Lill-1, Lill-2, Lill-3, + rolebased ‘numbers’ ‘vakt-kirurg’, vakt-anestesi’ etc. • Alt. 1: 3 diff. ringing signals on one device • quit call on level 1 if a level 3 arrives etc. • or something like ‘call waiting’ • Separate role device (?) or integrated on the same device ( St.O. solution?) • Alt. 2: 3 different devices + a role based device(more like today ‘array of responsibilities’ / medalje-fasaden) • manual toggling of the different devices (easy!) • Scholl et al: And easy to hand over a role device to another person

36. CSCW issues(4: use of general IM/phone) • ‘Standard requests’ for calls (i.e. formatted, Dahl et al, 2004) • “The query from the nurse is in form of a standard request for an assessment. The context of the assessment consists of an identification of patient Palmer, and the relevant part of his medication plan for Warfarine that nurse Neil was studying on the MEPC when sending the request.” • Can IM + Shared applic. + EPR be used instead? Consider the following scenario using Unified collaboration (UC): • Nurse Ann looks up in EPR at PC (pas. terminal or elsewhere) • Nurse Ann looks up in the buddy list, chooses available Dr. Eve • IM and or Phone + video (completely standard solution, existing SIP tech.) • Ann initiates ‘share my patient in EPR’ • This needs a small UC integration in order to send this info to Dr.Eve, and have Eve start her EPR system with this right patient (Eve must log in etc.) • Eve now view her own version of EPR and may write etc. Alt.: 4’ Ann initiates ‘share my patient view via video’ 5’ Eve does a ‘remote view’ of the view of Ann no logging of this joint view (quite similar to sharing a view on a PC in a meeting today) 6’ Eve may not write in the EPR, if that is needed she needs to move

37. CSCW 4: Mediated communication/life threatening? • Adding session handover and hence increased multimedia should allow for more ‘local telemedicine conferencing’ as described in Scen. VI. • Hence reducing walking (?) for Dr. Eve • What about the impact of the cure and care? • Today new ICT in hospitals may be introduced without involving reg. ethical. com. (‘operation’, not research) • See ‘continuity and communication problem’ in health care (A. Hafstad, Aftenposten, 3.3. 2008) • Phone calls have shown to be involved is some cases (patient 4) • But maybe the continuity problem is bigger?

38. CSCW, situatedness or generality • From this study we see clearly that Telenor wants general solutions ’fitting everywhere’ (or nowhere!) • Clear conflict between situated context and generality • However: In our implementation we are separating quite clearly between ’computing context’ and ’social context’ (the latter we do not model, leave to the user) • However a changes ’social/mental context’ during the call may change the now wanted media types away from the (stored) initial media types wanted. • This cannot easily be detected

39. Thank you!Questions? lillk@item.ntnu.nowww.item.ntnu.no/~lillk

40. Leftovers • 2 scenarios (from business and health care) • Background work and theories • Enabling technologies (SIP, SD, loc. etc.) (short) • High Level Requirements for the application ENME • Overview of implementation • Discission in a health care perspective (CSCW, HCI rel.) • Final notes regarding STS • example of the ubiq. awareness system for nurses • More leftovers

41. The case of patient and presence signals • Every system is used in a social context, never used according to designers intent or end users intent! • Example: • Nurses are intended to press the presence button (‘tilstede’) when visiting patients, causing the lamp to turn green (other causes as well regarding sound alarms etc.)

42. State Q:Quiet State A-N:(Active, normal) State P:PresentHandled (’tatt klokka’) State:A-HighActive Alarm f highest priority Everything quiet in the room, no active signals regarding this room This indicates that a patient has called for help. (lamps blinking) This indicates the presence of some (unidentified) nurse in the given room.(lamps lightned)It is activated (and deactivated) by pressing the button ‘tilstede’ Alarm! Heavy sound and heavy blinking. This level signals for a (still unidentified) nurse that ‘I need urgent help’. Nurses and doctors will run to the scene (i.e. to the room indicated). States in a room and ’meanings’

43. Alternative classification:more like the info displayed • Importance • O (void) • N (normal) • H (high!) • Nurse present? / who shall upgrade level • * No * Yes patient can upgrade to N • * No * Yes (deact.) nurse shall upgrade to H • * No * Yes no further upgrade? (maybe katastrofe-alarm?)

44. Not using ’presence’ (State P) (actively avoiding a functionality) • Often nurses actively cancels their own presence while entering the room (pressing twice at ‘tilstede’) • This seems to have 2 reasons: • Avoids forgetting to deactivate their presence if they need to run out of the room for some reason (causing patients to listen to alarms of level 3 (maybe also level 2) intended for the ‘present nurse’ • Avoids that another patient in the same room can activate a ‘double ring’, i.e. an alarm at level 3 • Level 2 is intended also to aid a nurse finding another nurse (lamp light), However, in this case level 2 will be very little in use and cannot be use to aid a nurse to find another nurse in case she needs help. • It would be interesting to read the original specification and intended use of level 2 (‘presence lamp’) and presence signal as a req. for activating double ring.

45. Nurse need ’some help’ • We may note that if a nurse needs non-urgent help (e.g. to lift a patient), she will not use level 3. Depending on the case she might take different actions, some of which are listed here: • A) If she knows in advance that she needs to lift a patient, she might find help first, and then enter the room. • B) She might go to the room, change to level 2 (nurse present), then realize that she need help, and the reset the level to 0 (by cancelling the presence button), and then initiate a new signal (i,e, a signal at level 1). • We may notice that in case B) there is no way for the system to see the difference between this signal (of level 1) initiated by a nurse, and an ordinary patient initiated level 1. There is no level 2,5 • Will a ‘level 2,5’ help in case B)? • Will identification and location tracking of all nurses help? (seems that this is common in US, but with some resistance) • Will PDAs to all nurses help? (or cause more interrupts?)

46. Alternative classification of levels • Importance • Level O • Level N • Level M (’2.5’) • Level H • Nurse present? / who can upgrade level • * No * Yes patient can upgrade to L • * No * Yes (deact.) nurse can upgrade to Medium/High • * No? * Yes nurse can upgrade to High • * No * Yes no further upgrade? (alt. katastrofe alarm?)

47. Health care scenarios (3/ ) • Studies of existing phone and paging practices (Scholl et al.) • Scen. X Dr. Andersen works at the oncology department. Sometimes she needs time to concentrate and work intensely on some patient results at the radiation section. At her section they have a specific role based phone which the nurses can use to contact the doctors. Dr. Andersen picks up this role based phone. She starts working. Then she gets a call from a nurse asking about a headache of a patient. She feels this to be an unnecessary interrupt. • This role based phone was termed the ’interruptor’, and always assigned to the most junior doctor

48. Interrupts? / Scenario (4/ ) • From their interviews with Doctors: • She (Dr. A) commented that she felt the phone was unnecessary and that it resulted in her constantly being “nagged” about small things related to patients such as a headache etc. that could easily be handled without contacting a physician. • (Scholl et al. did not interview nurses)

49. Requirements of ENME (details 1) • The ENME service shall be a value added service on top of an all-to-all telephony service. It shall work with a major standard MMoIP protocol. • 1.1 Caller and callee shall not need to be subscribing to the same telephony provider. • 1.2 The value added service ENME shall work with SIP as the underlying signalling protocol. • 1.3 Only one of the parties (caller or callee) need shall need to subscribe to ENME, the other session party can be an ordinary (multimedia) telephony subscriber. The non-ENME subscriber shall not need any special software or device to participate beyond standard based MMoIP equipment. • 1.4 Relevant standards shall be utilized also for the realization of the value added service (to the extent possible). All fully implemented

50. Requirements of ENME 2 (details 2) • ENME shall detect when a more suitable context occurs (i.e., fitting the proposed media types better). ENME shall then suggest an upgrade of the media types by moving the session to the new terminal(s). • 2.1 Some entity in ENME shall keep track of the context (such as wanted media types, and the currently available media types in the zones near the ENME subscriber) • 2.2 ENME shall not rely on the telephony software on the endpoint to store the initial media proposal.[1] • 2.3 A human user shall be involved before the media is changed. If the user agrees the rest of the transfer shall be automatic. • 2.4 This human user shall be the ENME subscriber, not the other party. All fully implemented[1] The entity on the endpoint may typically be a SIP UserAgent, and there is no guarantee that an arbitrary implementation of UserAgent will store this information.