Building Intelligent Dialog Systems to Promote Health Susan McRoy

1. Building Intelligent Dialog Systems to Promote Health Susan McRoy University of Wisconsin-Milwaukee May 2016

2. Outline Benefits of Intelligent Dialog Systems Our Recent dialog-related research • Question answering • Question & discourse classification Other recent research • Frameworks and formative studies for improved software designs for informal caregiving

3. Benefits of Intelligent Dialog Systems • Health providers and consumers often need to obtain complex new information to make the best choices. • Such information may be available from many sources and finding the most helpful information can be difficult. • It is also difficult for providers of health information to create documents that serve a range of information needs, without overwhelming any individual, whose own need is very specific. • Intelligent dialog systems can adapt the content they serve and how they serve it on the basis of what is known about the user's background, their decision task, or a history of their interaction with a system. • The key benefits are to address a diverse set of information needs by dynamically adapting the interaction.

4. An example where an IDS can help • Women who are pregnant may wonder what they should eat or not eat, what sorts of activities are safe (or unsafe) and when they should worry about various pains, swelling, or other physical conditions that are unusual for them. • They may have these information needs 24/7. • For many, searching the internet is not their preferred choice because there are so many documents that might be relevant, and many cannot be trusted, because anyone can post on the WWW. Even documents posted by trustworthy sources sometimes have a reading level that is too high for the target audience. • Health provider organizations and public health departments can and do put up more web-pages but many are going to get overlooked. • An IDS, provided as a web service, a SMS mobile service, or an installable app can be used to deliver a tailored question answering system with content vetted by health professionals.

5. Other examples • Giving users access to information from their electronic health record or a personally controlled health record. • E.g. to help people with chronic illnesses remember tasks from their discharge instructions and be able to track possible changes in their health status to catch serious changes early. • Providing customized information related to various health screenings, counselling, or preventive medications. • The US Preventive Services Task Force now has 95 recommendations for providers and 79 guides for consumers; they are targeted to specific ages, gender, but one could also consider health conditions for which personal risk may differ.

6. Our recent dialog-related research • We have used computational approaches to natural language dialog to support the practice of Public Health in the city of Milwaukee • Public health is • "the science and art of preventing disease, prolonging life and promoting health through the organized efforts and informed choices of society, organizations, public and private, communities and individuals" (Winslow, 1920)

7. Two target QA tasks • Provide information to improve birth outcomes • Milwaukee has a very high rate of fetal and infant mortality. • Many such deaths are considered preventable – e.g. by changes in moms' health behavior (smoking, nutrition, seeking help). • Provide information to explain complex choices related to prostate cancer screening. • Each year > 200,000 men are diagnosed and about 32,000 will die. • However, routine screening does not improved outcomes. CDC says screening decisions should be shared & consider individual risk • Both these tasks create a diverse, but highly individualized sets of information needs – QA might help, by focusing on needed information.

8. The approach: 2 way automated text- messaging • Text-messaging can be a good way to reach many people: • Many people across all ages and ethnic groups now use messaging (For the latest numbers, see Pew research) • We found typically used WWW sources often have > 11th grade reading level, while MHD clients typically have a 5th grade level • In our surveys: • Very few subjects reported using the web for health info (either pregnancy or prostate cancer) • Among the older men from low-income areas of Milwaukee, slightly more reported using text messaging (56.4%) compared the Internet (47%) and only 37% had internet at home

9. Authoring Tool Content Dialog Strategy Dynamic User Model Dialog Manager Dialog History Manager Language Manager A Reusable Architecture for NL Dialog on low-cost devices G A T E W A Y

10. Dialogs to Improve Birth Outcomes • This project used automated question-answering on low-cost cell phones to increase Milwaukee mom’s awareness of factors related to healthy pregnancy and childcare. • It was conducted as part of the Public Health Impact Initiative, at the UWM Zilber School of Public Health.

11. User Study • We built a text database of answers targetted to the needs of low-income moms with the help of City of Milwaukee Health Department (MHD). • Surveys and focus groups were used to select topics. • We created software to map the stored content onto a logic for controlling a question and answer system that can be accessed either via email or SMS messaging. • For the assessment, we recruited women who were MHD clients to use the system for a period of one month each; 20 women completed the study.

12. The NL Dialog Engine At runtime, the dialog engine uses a combination of NL processing techniques (phrases, semantics, etc and not just keywords) to find the best match between stored questions and the user's information need Answers can also be customized, if desired, using patterns that whose values are retrieved dynamically from a separate database.

13. Results of the study • The 20 women who completed the study, generated a total of 291 unique questions, corresponding to 161 different topics. • About 46 % of the questions were covered by topics in the database • 26 % of these were matched correctly within the allowed time; • 58% of these were unmatched; • 1 % were mismatched and • 16% attempted matching exceeded the time allowed.

14. SMS-Expressed Topics • 1. Abdominal pains/ cramping (4.1%) • 2. Contractions/labor (3.4%) • 3. Baby movement (3%) • 4. Breast feeding (2.7%) • 5. Heartburn (2.4%) • 6. Pelvic pressure/pain (2.4%) • 7. Eating enough (2%) • 8. Morning sickness (2%) • 9. Prenatal vitamins (2%) • 10. Swelling (1.7%)

15. Key Lessons • The information needs of subjects and the forms they used for expressing them were very diverse. • Thus, a dialog-based approach, which can provide a tailored response to each person’s needs would be beneficial. • Methods for matching must be fast and precise, with extra care given to focusing on the main part of the sentence and the expected type of answer, since there are often contextually redundant words and much overlap among the words.

16. Feasibility of using IDS to support prostate cancer screening decisions To support cancer education, we explored approaches to assessing information needs and performing language recognition which would be needed to create an IDS. • One approach involved collecting consumer's questions posted to Community-based Question Answering web sites like AskMD. • We also conducted observational studies, including one where men could send questions via SMS and a MHD nurse provided the answers at the backend. • This followed surveys and lab-based observations to assess whether the modality would be likely acceptable.

17. Our Taxonomy of Expected Answer Types • Our taxonomy for cancer questions has 3 levels. • The top level includes 3 types: • Factoid • Patient Specific • Non-medical • These types correspond to the three possible ways that an automated system should respond: • Provide a direct answer • Explain why the answer must come from a health professional • Explain that the answer is outside the domain of the system For more fine-grained classification we developed a 10 category taxonomy on providing meaningful responses for the intended use.

18. Lower levels of EAT Taxonomy Factual (F) Patient Specific (PS) Definition (D) Patient Diagnosis (PD) Patient Outcome (PO) Entity (E) Patient Recommendation (PR) Entity Explanation (EE) Patient Explanation (PE) Numeric Property Value (NPV) Non-Clinical (NC) Reference (R)

19. Phase 1Results • Our corpus contains about 1200 cleaned and coded questions (about 3% of those that we collected). • However, the distribution among the various categories was found to be very uneven. • Currently the intercoder agreement is best for the toplevel classification (factual, patient-specific, non-cliical), which was around .7 • To improve the distribution, we merged some categories in the taxonomy. • We also explored methods for dealing with unevenness.

20. Filtered Class Distributions

21. Classifier Testing 4 Classification Algorithms were tested • Multinomial Naïve Bayes (MNB) • Naïve Bayes (NB) • J48 Decision Trees (J48) • Sequential Minimal Optimization (SMO) 12 Corpora Modifications were tested (6 shown) • Unmodified (U), Spelling Correction (SC) Threshold Term Reduction (T5, T25), Range Term Reduction (R30, R50) But, best performance was achieved only by using weighted classification and resampling, which yielded accuracy of F1= 0.963

22. Human Subjects Studies related to Men's QA • We did 3 types of studies: • Written surveys • A lab-based study where men were asked to obtain information on prostate cancer either using a search engine or by forming and sending a text message with their questions. • A field study where participants could text to an email address monitored by a health department nurse.

23. Results of 2 studies • In surveys, lab and field studies, men reported preference for using text messaging to web-based search. • Subjects in the lab study were as successful as those using search (in terms of number of tasks completed) even though they were using an unfamiliar cell phone and the cell service was often slow (we used the lowest-cost prepaid svc). • In the field study, subjects’ questions were factual, but distinct. One subject sent an average of 3 questions per day – but most averaged 1 or 2. A few men sent no questions, but said “ok” or “thank you”.

24. Some other projects • We have been investigating new health management systems for informal caregivers in the home. We are interested in how targetted feedback and communication can help improve user acceptance. • Completed work includes studies of physical ability and attitudes among older caregivers. • We have also created a framework for predicting the usability and long term use of software and have been conducting user studies to see how well it aligns with people's assessments.

25. Summary • My research spans a broad range of areas, including natural language processing and intelligent user interfaces, and usability studies. • Recent applications involve both public health and health care. • Joining me in this work have been faculty from Communication, Nursing, and Health Sciences as well as students from computer science, biomedical and health informatics, and biomedical engineering.

26. References McRoy, S. Jones, S. and Kurmally , A. “Toward automated classification of consumers' cancer-related questions with a new taxonomy of expected answer types”, Health Informatics Journal, 2015. McRoy, S., Cramer, E. M., and Hayeon Song, “Assessing Technologies for Information-Seeking on Prostate Cancer Screening by Low-Income Men”, Journal of Patient-Centered Research and Reviews, 2014 1(4): 188-196. Song, H., May, A., Vaidhyanathan, V., Cramer, E. M., Owais, R. W., and McRoy, S. “A two-way text-messaging system answering health questions for low-income pregnant women”, Patient Education and Counseling, 2013 [PMID: 23711635] McRoy, S., Vaidyanathan, V., May, S. and Song, H. “An Open Architecture for Messaging-Based Consumer-Health Question Answering”, Proceedings 2nd ACM SIGHIT International Health Informatics Symposium (IHI2012), January 2012. Prasad, R., McRoy, S., Frid, N., Yu, H. and Joshi, A., “BioDRB: The Biomedical Discourse Relation Bank”, BMC Bioinformatics 12: 188, (18 pages) 2011