Mobile Dictation With Automatic Speech Recognition for Healthcare Purposes

1. Mobile DictationWith AutomaticSpeechRecognition for Healthcare Purposes Tuuli Keskinen, Aleksi Melto, Jaakko Hakulinen, Markku Turunen, Santeri Saarinen, TamásPallos TAUCHI research center, School of Information Sciences, University of Tampere, Finland Riitta Danielsson-Ojala, Sanna Salanterä Department of Nursing Sciences, Faculty of Medicine, University of Turku, Finland Kites symposium 2013

2. Content • Background & Motivation • Dictationapplication • User evaluation • Results • Discussion and conclusion • Endingwords

3. Content • Background & Motivation • Dictationapplication • User evaluation • Results • Discussion and conclusion • Endingwords

4. Background • Firstspeechrecognitionsystems for medicalreportingweredevelopedover20 yearsago[1] • Doctors’ dictationsarestillcommonlytypedmanually, bututilization of speechrecognition is increasingespecially in radiologyand pathology • Nurses’ use of speechrecognition is rare and oftenlimited to filling the templates [X] Numbersrefer to the actualreferences in the paper.

5. Background • Utilizingspeechrecognition in Finnishhealthcarestudied, e.g., in [2] whereradiologistswerefollowedchangingfromcassette-basedrecording to speechrecognitionbaseddictating • Severalstudies in the area of speechrecognition in healthcaredone, e.g., [1, 3, 4, 5] • Previousstudiesfocusmainly on objectivequalities, such as dictationdurations and recognitionerrorrates

6. Motivation ”Voi kun meilläolisimahdollisuus saneluun!” - Anonyymi YTHS:nsairaanhoitaja

7. Motivation for ourstudy • Paucity of utilizingspeechrecognition in Finnishhealhcare, especially in nursing • Obvious and unnecessarydelays in gettingpatientinformation to the nexttreatmentsteps • Lack of researchfocusing on the userexpectations and experiences of dictationapplicationsutilizingspeechrecognition in healthcare

8. Content • Background & Motivation • Dictationapplication • User evaluation • Results • Discussion and conclusion • Endingwords

9. Dictation application • Based on ”MobiDic” by Turunen et al. [6] • The mobile client (Androidapplicationon a tablet) includesfunctionality for recording and editingdictations, and modifying the dictationtexts • The server side manages the dictations (audio and text) and communicates with speechrecognitionengines and M-Filesdocument management system

10. Dictation application • Notonlyspeechrecognition is utilized, but a variety of othertools is included to improveresults: • State of the artnaturalprocessingtools (e.g., spelling and grammarchecking) • Statisticsbased on useractions • Optimizedmultimodaltouch-screen U • Distributed applicationmodelmakes a variety of usecasespossible: • Real-timedistributedassisteddictation • Workflow management • Plug-and-play component management (e.g., speechrecognizer, NLP tools, document management) • UI canbeadapted for differentusagecases and devices

11. Dictation application

12. Dictation application – v2.0

13. Dictation application – v2.0

14. Dictation application – v2.0

15. Content • Background & Motivation • Dictationapplication • User evaluation • Results • Discussion and conclusion • Endingwords

16. User evaluation • Real-worldcontext, realusersand realdictations • Twowoundcarenurses in one of the UniversityHospitals in Finland • Lastedthreemonths in total, covering 30 and 67 dictations for the participants • Wizard-of-Ozapproach • The medicallanguagemodelavailablewasbased on medical and nursingdocumentation, and thus, itwasnotsufficient to recognize the languageusedby the woundcarenurses

17. Dictation application

19. Methodology • Backgroundinterview • Main focus on participants’ normalpractices on makingand/ordictatingpatiententries • Subjective data gathered with questionnaires • User expectations and experiences (SUXES [8]) • Usability-relatedexperiences (SUS [9]) • Open questions • Log data • Allapplication and servereventslogged

20. SUXES method • Enablescomparisonbetweenuserexpectationsbefore the usage and userexperiencesafter the usage on a set of statements • Expectations reported by giving two values • acceptable level: the lowest acceptable quality level for even using the system (or property) • desired level: the uppermost level that can even be expected of the system (or property) • Experiencesreported by giving a single value on the samestatements • Expectationsform a gapwhere the experiencedlevel is usuallyexpected to be • Ifbelow something is wrong; Ifabove  success

21. SUXES method • Expectations • Experiences • Comparison Low High x x Using the phone is fast. x Using the phone is fast. Using the phone is fast.

22. SUXES method • Expectations • Experiences • Comparison Low High x x Using the phone is fast. x Using the phone is fast. Using the phone is fast.

23. SUXES method • Expectations • Experiences • Comparison Low High x x Using the phone is fast. x Using the phone is fast. Using the phone is fast.

24. Expectations and experiences • Weused the nineoriginalstatements of SUXES • speed, pleasantness, clearness, error free use, error free function, learning curve, naturalness, usefulness, and future use • …and fiveadditionalstatementscomparing the dictationapplication to the normallyusedentrypractice • faster, more pleasant, more clear, easier, and prefer in the future

25. Content • Background & Motivation • Dictationapplication • User evaluation • Results • Discussion and conclusion • Endingwords

26. User expectations on the application Median responses of acceptable – desiredlevels(greyareas), n=2.

27. User experiences on the application Median responses of acceptable – desiredlevels (greyareas) and experiences (blackcircles), n=2. P1 and P2 refer to participant 1 and 2.

28. User expectationscompared to normalentrypractice Median responses of acceptable – desiredlevels(greyareas), n=2.

29. User experiencescompared to normalentrypractice Median responses of acceptable – desiredlevels (greyareas) and experiences (blackcircles), n=2.

30. Content • Background & Motivation • Dictationapplication • User evaluation • Results • Discussion and conclusion • Endingwords

31. Discussion • The desired level was 6 or 7 on all statements • The experiencedlevelwas at least 6 on allbutonestatements • The usefulness of the dictationapplicationcanclearlybeseen in the results • Moreimportantly, the participantswouldpreferusing the application in the future, i.e., theywouldbeready to droptheirfamiliar and saferoutines

32. Conclusion • Due to nothaving an accurateenoughlanguagemodel for nurses’ purposes, weused a Wizard-of-Ozscenario to finalize the speechrecognitionresults • The userexperienceresults show a truepotential for ourdictationapplication – notonly to smoothendictationprocess, but as a relevant option for writing the nursingentries

33. Futurework • Finalizing a languagemodel for nurses and utilizingit in Finnishhealthcare to enabletotallyautomaticdictation-to-textprocess is crucial • Wearenotdeveloping the languagemodels by ourselves, butwillbe in closecollaboration with ourpartnersin the development and evaluation • Wearealsodevelopingourapplicationfurther to provideevenmorepleasurableuserexperience and seamlessprocess

34. FutureWork • In order to makethisreality, weneed a properprocess for iterativedeployment: not a stand-aloneproductwhichcanbesold to hospitals, for example • Wehavedevelopedallnecessarycomponents: client and backend software, connections to 3rd party components, tools to supportdeployment, and a completedeploymentprocess • Ready for commercialization – looking for partners!

35. Globalmarket

36. Globalmarket

37. Acknowledgements • Project ”Mobile and UbiquitousDictation and Communication Application for MedicalPurposes” (”MOBSTER”) • Fundedby the FinnishAgency for Technology and Innovation (TEKES) • Lingsoft and M-Files, and otherprojectpartners