Conformance Testing of Interoperability

1. Conformance Testing of Interoperability in Health Information Systems in Finland Medinfo 2007, Brisbane Wednesday, 22 August, Session S103, 11 AM Tanja Toroia, Anne Eerolaa, Juha Mykkänenb University of Kuopio, Finland a Department of Computer Science b Health Information System (HIS) R&D Unit

2. in this presentation • Introduction and background • Survey of conformance testing of interface specifications in healthcare applications in Finland • Findings of the survey • Recommendations based on the results of the survey

3. introduction:conformance testing of interfaces • Conformance testing = testing to determine whether a system meets a specification or not • Interface (standard) specifications are central for interoperability • Conformance testing of interfaces includes • conformity of applications and interfaces to standards • testing of the interfaces between applications • Does not cover testing of the non interoperability-specific features of the applications

4. background of the study • In most healthcare facilities, interoperability and safety require attention on cross-application level • Integration, distribution, security and interoperability emphasized lately in software testing • Testing related to interoperability • focus on deployment and integration phases of systems lifecycle • standards compliance • certification • Conformance to specifications a key area in interoperability • However, few / no empirical studies on the level of conformance testing in software companies in healthcare and in health service provider organizations • Survey: how much and how often conformance testing is performed in relation to the interface specifications in healthcare applications

5. main research questions • How can customers, i.e. hospital districts, influence the conformance of the software they acquire? • How much conformance testing do software vendors perform and how often are standards used? • What are the main goals of, problems with, and opinions about external interface testing services?

6. the survey Topic in this talk • Themes • General software testing practices • Conformance testing and standards • The organization of interface testing services • Web-based questionnaires with open and structured questions • The questionnaires are shown on http://www.cs.uku.fi/~ttoroi/questionnaire.html

7. sample and responses • Healthcare software vendors in Finland • 27 software companies which belong to a Health and Welfare IT R&D cluster in Finland (response rate 52%) • responses focused on companies producing EPR and departmental systems + consultancy services • Customers of the software vendors • 20 Finnish hospital districts, all mainland public healthcare districts (response rate 75%) • respondents mostly from specialized care facilities

8. results - Conformance testing and standards • Term 'conformance testing' was misunderstood by two respondents • 1 - 4 new versions each year for most applications • Interoperability standards and open interfaces used widely • Conformance testing performed variedly for entire applications and between specific systems • Demand for the use of UML models and XML in standards • Reported problems: • In some situations, no choice but to accept evident errors in applications or new versions • testing among the first areas to be "cut down" in some companies • often difficult to apply and interpret standards, also national specifics and laborious implementation hinder the adoption

9. results - Organization of interface testing services • Suspiciousness of external interface testing services, lack of knowledge about such services • Interoperability challenges not focused in testing • "testing services cannot help in integration" • Certification of application interfaces was not considered very necessary nor useful • Respondents were not familiar with the internationaldevelopments of interoperability testing or certification • e.g. IHE Connectathons, CCHIT, European certification initiatives

10. discussion and recommendations 1 • Perform interoperability conformance testing more rigorously • surprisingly missing link between quality requirements and testing, more time for testing and various conformance testing activities required for diverse requirements • Utilize open interfaces and use interface testing services • open interfaces are steadily increasing, promoting cost effectiveness in general and in testing activities • Provide proper skills and knowledge • needed: education about the benefits of standards-based interfaces, common definitions and policies for interoperability testing

11. discussion and recommendations 2 • Enforce the recommendations by authorities • respondents were in favor of official recommendations in relation to interoperability, security and usability • Organize neutral testing services • little market possibilites for commercial testing services • possibilities: automation of conformance testing • neutral services / national mandates need further exploration • Reuse testing experiences and information • some encouraging sharing of experiences from system introductions, could be extended to expert groups • Improve the testability of specifications and standards • conformance requires accuracy and quality from specifications

12. concluding remarks • Results can not be directly generalized • Standards: perceived necessary but burdensome • Conformance testing: hindered by attitude problems and lack of knowledge • Improvements needed in relation to demand, investments and standardization • But: • Interoperability testing is already part of common policies • National / large-scale initiatives draw attention to interoperability and shared requirements, solutions and services • Further discussions and developments encouraging (national healthcare IT services and related certification, interest and participation to interoperability testing on international level) • first author: Tanja.Toroi@cs.uku.fi • speaker: Juha.Mykkanen@uku.fi

13. other results from the SerAPI project in Medinfo'07 • Model-Centric Approaches for the Development of Health Information Systems (Tuomainen et al.) • A National Study of eHealth Standardization in Finland - Goals and Recommendations (Mykkänen et al.)

14. THANK YOU Tanja Toroi, tanja.toroi@uku.fi, Anne Eerola, Juha Mykkänen, juha.mykkanen@uku.fi, This work is part of the SerAPI and AVOINTA projects: www.serapi.fi/ www.centek.fi/avointa/ Projects are funded by the Finnish funding agency for Technology and Innovation TEKES together with a group of hospital districts / cities / joint municipal boards for healthcare and companies.