HCMDSS Workshop Certification & Requirements Working Group Report (WG 6)

1. HCMDSS Workshop Certification & RequirementsWorking Group Report (WG 6) June 2-3, 2005 Philadelphia, PA John Hatcliff, Kansas State University (hatcliff@cis.ksu.edu)

2. Peter Coronado -- Varian Medical Systems Steve Dimmer – Calypso Medical Technologies Jacob Flanz – Mass General Hospital Elsa L. Gunter -- University of Illinois at Urbana – Champaign John Hatcliff -- Kansas State University Mark P. Jones -- Oregon Health & Science University (OGI) Paul L. Jones -- FDA/CDRH/OSEL Brad Martin -- National Security Agency Rosemary C. Polomano -- University of Pennsylvania School of Nursing Stacy Prowell -- University of Tennessee John Rushby -- SRI Rick Schrenker -- Massachusetts General Hospital Oleg Sokolsky -- University of Pennsylvania Participants

3. State of the Art-- What We Do Well • Develop and approve stand-alone medical devices based on mature technology with moderate complexity • well-understood domain with a lot of experience • we have a lot preceding examples with a lot of experience • we know how to gather requirements, code, test, do hazard analysis, etc. in a reasonably good way • Advanced development techniques such as model-driven development and software product-line architectures are working well in specific device domains (e.g., Guidant pace makers) • Established mechanisms for reporting, tracking, and publicizing device related “incidents” (analogous to reporting air plane crashes) • Light-weight formal methods (e.g., ESC Java) have been effective in security evaluations • in non-regulated or certified settings, tremendous progress has been made (Intel, Microsoft, etc.)

4. State of the Art—Challenges Within Bounds of Existing Technology • We know how to… • test, write requirements, prototype, etc. • but how we can do this more accurately, in a more automated way, and achieve greater cross-leveraging of artifacts • Large-scale, complex devices still pose challenges • e.g., proton therapy facility • 100,000’s of validation procedures, test cases • …validation burden (time/costs) slows time to market

5. State of the Art—Challenges Within Bounds of Existing Technology • System integration for devices based on existing technology • We are doing pretty well with integrating products developed by a single manufacturer, e.g., Varian (Linac, connected to imaging device, treatment machine) • But there are clinical cases where people are incorporating devices from other manufacturers (currently tech people must be trained to recognize potential incompatibilities) • even something as simple as incorporating tubing into pump (which were both approved separately) malfunctioned in clinical setting

6. State of the Art –The “Heroic” Clinical Engineer • In general lack of a systems view or systems model that would enable one to reason about clinical environments, processes, and ad hoc integration of devices • “Clinical Engineers” are faced with the task of deploying collections devices in ad-hoc ways in constantly changing devices • some devices interact directly • some device interact through patients • some devices with even looser coupling • research on formalizing clinic environment, medical process that accommodates variability across different environments • Devices being delivered with little or no, or inappropriate documentation (especially integration documentation) • Incorporate inter-patient variability of (e.g., very obese patients) in requirements for monitoring devices, etc.

7. Research Challenge --Requirements Research • More automated and rigorous manner for eliciting, capturing, and leveraging requirements • Support semantics querying or simulation of use cases directly from requirements • More effective automated derivation of test cases from requirements • Automated generation of visualizations and natural language description from requirements

8. Modeling/Formalization of Clinical Environments and Processes • Trend: Proliferation of sophisticated devices in complex environments where exceptions are the norm • Clinical engineers currently act as informal “system integrators” of devices • Must develop a more rigorous approach to capturing, visualizing, reasoning about clinical environment & processes • How to incorporate expected variability within clinical environments? • Can we mine these rigorous descriptions (e.g., of processes, workflows, etc) to guide device requirements development or detect problematic device interaction within a clinical environment? • Can we use these models to clarify goals and focus the tasks to achieve more rigorous clinical validation?

9. Re-orienting Current Procedures for Component-wise Certification • Components as commodity • framework that creates a business environment that produces a variety of re-usable components are marketed • Certification effort should focus on validity of composing components • certification artifacts should be reused (integrator should not have the burden of re-certifying component internals) • want to be able to synthesize certification of assembly from certification of components • Recognize the differences in integration scenarios • Commodity components purchased vendor to produce a plug-n-play system with a suite of components • Components purchased by hospitals and assembled by clinical engineers (should their be any oversight of such engineering?) • Requirements that capture all important aspects of interfaces • functionality • quality-of-service • safety/faults • Elevating methods for, e.g., protocol validation to a system of systems level • Inspiration from but looking beyond RTCA Integrated Modular Avionics

10. Certifying In the Presence of Change/Variation/Evolution • Frequent software patches • Move from “software product lines” to “certification product lines” • FDA recognition of tools the perform impact analysis and traceability to reduce the effort in retesting/re-certification • “Make” for certification • More sophisticated tools for managing certification artifacts and for automatically detecting what things that need to be rework in the presence of change

11. Problems with Existing Technology • Reporting, tracking, and publicizing, and interpreting device “malfunctions” (distinct from “incidents”) across manufacturers is problematic • need mechanisms for reducing the burden of re-certification • impact/change analysis tools, having FDA recognize artifacts produced by such tools • some devices incorporate “flight recorder” technology (e.g., Varian Linac) • open up certification resources for looking at bug-fixes, evolution of software

12. Dealing Effectively With Security Certification Issues • Trend: desire to integrate a variety of device types across a variety of network infrastructures • Network types: • national, regional, local, home, body-area (wired, wireless), • Trend: Blending of medical devices and medical informatics systems • Devices that stream data (pace makers) into medical records that may be mined for broader clinical studies • Trend: Competing concerns • security, availability, … • competing quality of service when devices deployed on the same framework (leading to the possibility of denial of life-giving services) • Need to investigate and vet security standards such as Common Criteria (most stringent levels require formal methods) and MILS architecture

13. Certification in Context ofData Communication • Ontology of medical information • Common language framework for exchanging a variety of forms of data • how does this relate to DICOM, HL7, etc.? • How do we certify that systems maintain the integrity of the data? • What’s keeping IEEE 1073 from being implemented?

14. Other Issues • Unique identifiers • Physical Integration Standards • Use of COTS software

15. Research Infrastructure • Academics will likely not be able to effectively address certification challenges with V & V technologies without a detailed understanding of certification process • Certification training for academics • Workshop on certification/approval process working with example artifacts • Mock “red team” certification for products/artifacts developed by research teams • …likewise, the need for effective communication of available V & V technologies to industry

16. Research Infrastructure • “Open Experimental Platform” for High Confidence Medical Devices • Funding agency pays for… • …vendors to develop representative examples of certified systems • …vendors to provide description of development process • …vendors to provide “challenge problems” to researchers • …vendors to provide “face time” to meet with researchers and evaluate developed tools against base-line development processes • One example OEP proposed by Peter Coronado (Varian) et. al.

17. Research Goals (5 Years) • Apply existing process modeling languages to model clinical environments and processes • Suite of sophisticated requirements tools… • requirements capture, simulation, querying • Paths to incorporating more tools into the certification effort (adding value) • Certification of development tools (analysis, traceability tools) for the purpose of reducing (re)-certification burden • Instrument conventional formal methods tools (static analysis, model-checking) to produce a variety of artifacts (test cases, natural language description of traceability steps) • Demonstration of pervasive use of model-based development techniques with automated reasoning for • component conformance to interfaces • component capability based on checking interface capability • end to end reasoning of system behavior • …emphasize management, integration, and automatic generation of certification artifacts – models form a substrate within which all certification-relevant artifacts are attached

18. Research Goals (10 Years) • Re-orienting the certification process toward a component-based certification • Certified components as commodities • Pervasive use of secure, QoS-aware, fault-tolerant, certified middleware • Integrated end-to-end model-based development frameworks dealing with composition, evolution, change • Effective demonstration (metrics, etc.) necessary to change industry / regulatory practice • Wide body of interoperability standards • More sophisticated compositional analysis tools • Tools for compositional hazard analysis