BioSTORM: A Test Bed for Configuring and Evaluating Biosurveillance Methods

1. BioSTORM: A Test Bed for Configuring and Evaluating Biosurveillance Methods Samson W. Tu, M.S.,1 Martin J. O’Connor, M.Sc.,1 David L. Buckeridge, M.D, Ph.D.2Anya Okhmatovskaia, Ph.D.2 Csongor Nyulas, M.S.,1 Mark A. Musen, M.D., Ph.D.1 1Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA 2McGill University, Department of Epidemiology, Biostatistics and Occupational Health, Québec, Canada Task-method ontology System Architecture • Implementation • We used (1) the JADE (Java Agent Development framework) to implement inter-task communication and the control structure, and (2) Java and the R statistical package to implement methods. • On start up, the Controller agent invokes the Configuration agent to instantiate JADE task agents based on the task-method ontology and an evaluation study configuration. Each task agent is configured with its associated method and waits for its inputs to become available on the Blackboard. • The Controller agent posts initial configuration data to the Blackboard to initiate the data-driven execution of the configured detection and evaluation algorithms Background Public health agencies in the United States have implemented hundreds of syndromic surveillance systems at a cost of hundreds of millions of dollars. Despite the accelerating enthusiasm for this approach, however, there are remarkably few published evaluations of outbreak detection through syndromic surveillance. To meet this need and building on our earlier work to create a scalable architecture for configuring biosurveillance methods, we are developing (1) an explicit representation of aberrancy detection algorithms and their properties to facilitate systematic comparisons and (2) a computational test bed that can draw on real-world data sources and that will allow users to configure, run, and evaluate alternative surveillance methods. The test bed will lower the barriers to evaluation tremendously Configurator agent Get-data agents Detection agents Evaluation agent Data Evaluation study configuration Results Blackboard agent Controller agent Monitor agent Biosurveillance tasks and methods • Task-method ontology and evaluation study configuration • We created an ontology of tasks and methods in Protégé OWL to define their properties formally. Tasks are defined by their inputs and output. Methods are specified in terms of their semantic properties, which are further characterized as configuration parameters, input data, or computed results. • To configure an evaluation study, an analyst (1) creates instances of tasks and methods using Protégé, (2) organize them into algorithms, and (3) use named variables to specify data flows and the binding of data to task inputs/outputs and to method properties. • An algorithm is composed of tasks, each of which has an associated method. A method is either primitive or further decomposes into a subalgorithm • Special iteration tasks and methods concatenate results of subalgorithms into vector-valued variables Evaluation studies Task structure of an evaluation study C-family algorithms tasks/methods Hutwagner L, Seeman GM, Thompson WW, Treadwell T. A simulation model for assessing aberration detection methods used in public heatlh surveillance for systems with limited baselines. Statistics in Medicine 2005;24:543-550. Aberrancy detection Sample Results Performance of C1, C2, C3 algorithms on selected CDC EARS data sets C-family algorithm evaluation study configuration Task-method ontology in Protégé OWL Holt-Winters generalized exponential smoothing tasks/methods