Long-term Usability Testing for Public Health Information Technology: BioSense 2.0

1. Long-term Usability Testing for Public Health Information Technology:BioSense 2.0 Amanda Recker Jamie Pina, MSPH, PhD Barbara L. Massoudi, MPH, PhD RTI International 2013 International Symposium on Human Factors and Ergonomics in Health Care: Advancing the Cause March 11, 2013

2. BioSense 1.0: Web-based syndromic surveillance • Mandated in the Public Health Security and Bioterrorism (BT) Preparedness and Response Act of 2002 • Nationwide integrated system for early detection and assessment of potential BT-related illness • 2003 Funding provided by Congress to CDC • Development of BioSense infrastructure started, initial focus on: • VA and DoD • Direct reporting to CDC of detailed clinical data by civilian hospitals • Began soliciting more limited data from health departments (HDs) that had already established automated systems for ED-based syndromic surveillance • By 2007, 8 state/local HDs connected

3. Recommendations from Prior Evaluations • Strengthen state and local public health engagement • Enhance state/local HD syndromic surveillance capacity • Increase participation of state/local HD syndromic surveillance systems (improve coverage) • Share data with HDs from hospitals reporting directly to CDC • Share governance with public health community • Leverage investments in electronic health records (EHRs) • GAO, 2008: Adopt an “open, distributed computing model” • Improve utility of the data and data sources • Preparedness role: Greater “all hazards” emphasis • Expand uses for broader spectrum of public health concerns

4. BioSense 2.0: Timeline • June 2010: Redesign begins • November 2011: Opened for business • November 2011 – June 2013: Onboarding new jurisdictions • 35 jurisdictions signed the Data Use Agreement (DUA) • 17 fully onboarded • April 2012: Retired BioSense 1.0

5. BioSense 2.0: Approach • Shift from a need-to-know to a need-to-share and co-create approach • User-centered design • Stakeholders engaged in every step of the redesign • HDs fully control “their data” at the level of granularity they choose • More options for data sharing with other jurisdictions and CDC • Alignment with ONC and Meaningful Use • Agreed-upon core syndromic surveillance data elements • Collaborations with public health professional associations • Funding to states: Meaningful Use syndromic surveillance adoption, build capacity, join BioSense 2.0 • Cloud technology: distributed, easy to adopt, cost effective, secure

6. Application Home Page

7. Encephalitis, Meningitis, WNV = CNS Inflammatory Disease

8. Why Long-term Usability Testing? • Long user-centered design lifecycle • Expectation management • Stakeholder ownership • BioSense 2.0 continuously changing and growing • Longitudinal usability testing • User satisfaction • Efficiency • Functions not changing: building a query, viewing results, analyzing the data, sending and saving information • Two approaches to testing

9. 1. User-Centered Design • Qualitative data collection methods • The user knows best • Test the right participants • Broad range of public health professionals • Public health generalist – less sophisticated users • Syndromic surveillance epidemiologist – more sophisticated users

10. 2. Activity-Centered Design • Quantitative data collection methods • Behavior vs. opinions • Based on empirical data • Time-on-task analysis • Mouse-click analysis • Pathway analysis

11. Basic Activities in Interaction Design • Establish requirements • Design alternatives • Develop prototype • Conduct evaluation (Rogers, Sharp, & Preece, 2011) (Rogers, Sharp, & Preece, 2011)

12. How to Choose Users • Interact directly with the system • Epidemiologists, state and local public healthprofessionals • Manage direct users – decision-makers • Public health directors • Use similar syndromic applications

13. Generating Design Alternatives • Activity-centered design • Morae software • Scenarios and tasks • Closed/open question response • Time on tasks • Mouse-click analysis • Pathway analysis • Critical incidents • User-centered design • Open question response • Focus groups • Expectation testing • SUS • Expert evaluation • Usability heuristics (Nielsen)

14. Scenarios and Tasks Scenario: “Over dinner at [a public health conference] an argument has erupted, but luckily as a BioSense 2.0 user you can settle this dispute. Health authorities in Virginia suspect that the flu season was more severe than it was in Michigan.” Task: “Please determine which state, Virginia or Michigan, had more cases of influenza-like illness (ILI) starting in October 1, 2010 through March 1, 2011.”

15. Pathway Efficiency Analysis • Time on task • Mouse-click analysis • Pathway analysis

16. Design Alternatives

17. How to Choose Among These Alternatives? • If one person says something is a problem, do you change the design? • Resolved conflicting alternatives • Conducted feasibility analysis • In the end…relied on face validity

18. What Happens After Design Changes? • User training through webinars and videos • Expert user testing • Focus group sessions • Continually comparing SUS scores

19. Future Activities • Incorporate eye tracking into testing protocols • Conduct on-site testing and evaluation • Task analyses of routine versus event surveillance • Information models for routine and event surveillance