NetCDF

1. 2010-2011 NEON Computer Science ClinicCommon Data Services for Ecological Data NEON Problem Statement System Design Testing NEON, the National Ecological Observatory Network, is a NSF-funded project to collect and manage data from across the U.S. NEON's mission is to enable understanding and forecasting of the impacts of climate change, land-use change and invasive species on continental-scale ecology -- by providing infrastructure and consistent methodologies to support research and education in these areas. This project uses JUnit to automate the testing of software deliverables. A set of unit tests cover the data-access API call, getData, as well as getConfig, a configuration-access API call. Both edge cases and common scenarios are considered. The majority of these tests use data from a test database to verify their correctness. To support these tests, the team has written and provided scripts to populate the test database. Ecological studies Community Environmental modeling / monitoring data products Data formatting / visualization Applications Data gap-filling / smoothing / aggregation Common Data Services Deliverables Data NEON's SQL database other data files • Thus, the team is delvering to NEON: • A hibernate-based system that traverses NEON's database and returns Java objects encapsulating data/configurations. • A netCDFHandler class that can write Java DataResult objects into NetCDF files. This file-creation is done lazily. • An overall CDS layer that thus offers NEON's application developers a consistent interface -- NetCDF files -- whether the data is stored in NetCDF or in NEON's database. If desired, developers can access the underlying Java objects. raw data from towers, scientists, others... NEON's mission is to make its large, varied datastores available to the scientific community. In addition, NEON application developers – and applications -- need the ability to access data systematically and without worrying about the details of the model by which the data is stored. • Thus, NEON asked the HMC clinic team to design and prototype a Common Data Services (CDS) software layer. The CDS provides application developers a consistent, extensible abstractionthrough which to access NEON's data, whether its database or files. The team has also documented and tested its final, deliverable CDS system. Given a query for data, the Common Data Services (CDS) layer provides a list of NetCDF files as its primary output method. If desired, the user can choose to access the “bare” data, provided as a Java object, instead of the file. NetCDF files generated by the CDS are lazily created: they aren’t generated until the user requests them via a function call. This saves time and disk access in the case that the calling application isn’t interested in them. NEON organizes the U.S. into 20 ecoclimatic domains. • The clinic team has also created a javascript/PHP interface to high- light an example of the type of application CDS might support. It uses historic CO2 data from Mauna Loa, Hawaii. Data Sources • NEON’s Computational Infrastructure (CI) systems will handle a wide variety of data from different soruces and formats: • The Fundamental Instrument Unit (FIU) monitors physical and chemical climate properties such CO2 and moisture. NEON is designing and deploying observation platforms to provide such data. Those data will be collected into NEON's large SQL database at its headquarters in Boulder, CO. • The Fundamental Sentinel Unit (FSU) collects specimens of local species and data on biodiversity and populations. • The Airborne Observation Platform (AOP) analyzes changing land use, vegetation cover, and species migration. • The Land Use Analysis Package (LUAP) compiles and assess historical data, much of which is in NetCDF format. Database Traversal NetCDF Example: getData( "Mauna Loa, HI",“Raw CO2”) NetCDF is a community-standard scientific data file format supported by a large number of existing applications and well-maintained libraries. Because the ecological community and NEON itself use NetCDF already, we chose NetCDF as our common intermediate data format. The Common Data Services layer outputs the results of a query in NetCDF form, whether the data are stored in a flat file or NEON's database. • The provided CDS system is necessarily a prototype, because NEON and its data handling policies are evolving during this early phase of its deployment. The clinic’s CDS system offers a flexible foundation layer to which NEON and other developers will add additional capabilities in order to monitor, maintain, and investigate the nation’s largest store of ecological data. Hibernate Hibernate is a Java-based object/relational mapping library that allows developers to treat database tables as objects. Thus, Hibernate simplifies the process of reading databases: instead of constructing complex SQL queries, the CDS traverses a collection of interconnected Java objects. In addition, we use the Hibernate Spatial and the JTS Topology Suite in order to access and manipulate geometric objects from the database. These libraries also offer a large set of transformations, such as polygon intersections and area computations, on arbitrary geometric objects. Acknowledgments NEON Liaisons RobertTawa DJ Spiess Team Members Jason Garrett-Glaser '11 Keith Ingram '11 (PM) Alejandro Lopez-Lago '11 HamsterBob Stewart '11 Faculty Advisor Zachary Dodds A NEON observation platform for collecting FIU data. The NEON data model, accessed through Hibernate