Hydrologic Information System for the Nation

1. Hydrologic Information System for the Nation Ilya Zaslavsky Spatial Information Systems Lab San Diego Supercomputer Center UCSD http://his.cuahsi.org http://hiscentral.cuahsi.org http://hydroseek.net http://river.sdsc.edu/ucsddash http://wron.net.au/DemosII/Modules/ODMKMLGatway.aspx http://maxim.ucsd.edu/mattsmaps/storet.aspx UCSD Seminar, February 23, 2009

2. SDSC Spatial Information Systems Lab http://spatial.sdsc.edu/lab/ Research and system development • Services-based spatial information integration infrastructure, CI projects • Mediation services for spatial data, query processing, map assembly services • Long-term spatial data preservation • Spatial data standards and technologies for online GIS (SVG, WMS/WFS) • Support of spatial data projects at SDSC and beyond In Geosciences (GEON, CUAHSI, CBEO,…) services In Neurosciences (BIRN, CCDB) In regional development (NIEHS SBRP, CRN…) Contact: zaslavsk@sdsc.edu

3. Consortium of Universities for the Advancement of Hydrologic Science, Inc. 122 US Universities as of July 2008 An organization representing more than one hundred United States universities, receives support from the National Science Foundation to develop infrastructure and services for the advancement of hydrologic science and education in the U.S. http://www.cuahsi.org/

4. What is the CUAHSI HIS? CUAHSI HIS: NSF support through 2012 (GEO) Partners: Academic: 11 NSF hydrologic observatories, CEO:P projects, LTER… Government: USGS, EPA, NCDC, NWS, state and local Commercial: Microsoft, ESRI, Kisters International: Australia, UK Standardization: OGC, WMO (Hydrology Domain WG, CHy); adopted by USGS, NCDC An online distributed system to support the sharing of hydrologic data from multiple repositories and databases via standard water data service protocols; software for data publication, discovery, access and integration.

5. Map for the US Build a common window on water data using web services Observation Stations Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System USGS National Water Information System NOAA Climate Reference Network

6. Water Data Web Sites

7. NWISWeb site output # agency_cd Agency Code # site_no USGS station number # dv_dt date of daily mean streamflow # dv_va daily mean streamflow value, in cubic-feet per-second # dv_cd daily mean streamflow value qualification code # # Sites in this file include: # USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC # agency_cd site_no dv_dt dv_va dv_cd USGS 02087500 2003-09-01 1190 USGS 02087500 2003-09-02 649 USGS 02087500 2003-09-03 525 USGS 02087500 2003-09-04 486 USGS 02087500 2003-09-05 733 USGS 02087500 2003-09-06 585 USGS 02087500 2003-09-07 485 USGS 02087500 2003-09-08 463 USGS 02087500 2003-09-09 673 USGS 02087500 2003-09-10 517 USGS 02087500 2003-09-11 454 Time series of streamflow at a gaging station

8. CUAHSI Observations Data Model http://his.cuahsi.org/odmdatabases.html

9. Water web pages Water web services Information communication Water Markup Language (WaterML) HyperText Markup Language (HTML)

10. Set of query functions Returns data in WaterML Water Data Services NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites)

11. Hydrologic Information System Service Oriented Architecture Test bed HIS Servers HIS Lite Servers Central HIS servers External data providers Global search (Hydroseek) Deployment to test beds Customizable web interface (DASH) Other popular online clients HTML - XML Desktop clients Data publishing HIS CentralRegistry & Harvester Water Data Web Services, WaterML WSDL - SOAP Ontology ETL services Controlled vocabularies Metadatacatalogs ArcGIS WSDL and ODM registration Matlab IDL, R Ontology tagging (Hydrotagger) Excel ODM DataLoader Programming (C#, VB..) Streaming Data Loading MapWindow ODMTools HISDesktop Modeling (OpenMI) Server config tools

12. Central HIS Data Services Catalog

14. Semantic Tagging of Harvested Variables

15. Hydroseekhttp://www.hydroseek.net Supports search by location and type of data across multiple observation networks including NWIS, Storet, and academic data

16. How we work with agencies on web service access to observational data • Establish an agreement with the agency on joint development of water data services, identify agency partners (ideally, with time to support joint work) • Identify the scope of the service, databases to be exposed, and access control, assign network and vocabulary codes • Map semantics of the service to WaterML semantics, and verify with the agency • Include discussion of agency data and interoperability issues in the context of OGC Hydrology Domain WG (if needed) • Develop a first draft of the web service • Unit testing, over a series of validation cases developed jointly with the agency • Harvest an observations metadata catalog for agency data, to be housed either at SDSC or at the agency • Develop a procedure for catalog updates • Register the water data service at HISCentral(including mapping of variables to ontology terms), and test it using HydroSeek and HydroExcel. Document the service • Review and test the service together with the agency, for possible approval as “operational”

17. Against the NIH Syndrome 2006: CUAHSI HIS web services are discussed on the BASINS mailing list as a new way to access hydrologic data. The list is mostly used by hydrologists and developers outside academia; NCDC develops ASOS web services following WaterML 2007: MOU with USGS; USGS is developing WaterML-compliant GetValues service; GLEON uses an early version of ODM to develop their own schema (VEGA); Phoenix LTER is developing ODM (in MySQL) and WaterML services (in Java); A Google Earth-based client for CUAHSI web services is developed at CSIRO, Australia; Deployment to 11 hydrologic observatory test beds, + CBEO (CEOP project) 2008: KISTERS develops WaterML-compliant web services over their database, for a client; MapWindow open source GIS develops WaterOneFlow parsers; Florida, Texas and Idaho use ODM and WaterOneFlow web services to provide access to state data repositories; New Jersey is considering the same; Another CEOP project, at UC-Davis, is implementing ODM (in Postgres) and web services (in Java); Stroud Water Research Center; WRON; AWI… many that we don’t know… Now SBRP: data from UCSD, UA, more?

18. Visualization and Analysis of Large Datasets Tiled wall (Falko Kuester) OLAP cubes for repositories (via Excel and online GIS) Google Earth client

19. Coming soon… All Hands Meeting of CUAHSI HIS at SDSC; April 6-7 2009 Meetings at BOM, CSIRO, global water initiative (second half of March) Collaboration with World Meteorological Organization (joint charter between WMO’s CHy and OGC’s Hydrology DWG) Open Geospatial Consortium: Hydrology Domain Working Group, focus on WaterML 2.0 Automated facilities for uploading/hosting observations data, CZO HIS capacity building workshop in Kuala Lumpur (plus a few workshops in the US) Integration with real-time (via DataTurbine), with hydrologic models (via OpenMI, CSDMS), animations, spatio-temporal interpolation Desktop HIS (this year)

20. Summary • Generic method for managing and publishing observational data • Supports many types of point observational data • Overcomes syntactic and semantic heterogeneity using a standard data model and controlled vocabularies • Supports a national network of observatory test beds but can grow! • WaterML is a standard language for consistently communicating water observations data from academic and government sources using web services • National Water Metadata Catalog is the most comprehensive index of the nation’s water observations presently existing Join the Water Data Federation!