Implementation through European Community Research Projects

Implementation through European Community Research Projects State of the art Technologies for GIS Management Systems Lykiardopoulos Angelos, Iona Athanasia,Lakes Vasilis, Balopoulos Efstathios, Kalkavouras Constantine Hellenic Center for Marine Research

Data management and Web Services Quality control in HNODC data base Poseidon system, a short description Sections of this presentation

At the end of 80’s and early 90’s several Oceanographic Data centers had created Databases with mass storage of spatial Information which concerned Oceanographic research outcomes. This was en enormous step of digital storage because big data volumes organized in Database Systems. Data Management and Web Services a Historical Brief PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

In other words the data centers owned databases Based on various RDBMS mechanisms With their own database Schemas With their own data vocabulary (if existed) With raw data in various formats With Applied various processes of data quality control (or with no data quality control) This effort leaded into the creation of several data bases without any Interoperability among them. Data Management Historical Brief PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

But the oceanographic community had the need of homogenized and comprehensive data sets for various regions and even world wide. Due to the limitations of information technologies this goal was achieved in the framework of several E.U. projects Central accumulation of data from various Data Centers took place Data sets were processed in common data formats and more or less same Q.C. procedures. Finally the dissemination of these Data sets was carried out mainly via digital media (CD-ROMs, etc). The centralized Data Management Idea… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

During this process: Diverse Data form a wide range of Data Centers were gathered. Passed common homogenization and QC procedures. And finally some comprehensive Data sets were produced and disseminated. But… The produced Data Sets were stand alone products stored in digital media. Without any dynamic mechanism of maintenance and enhancement. And without any online access. The centralized Data Management Idea… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

Thus, the idea of a distributed Oceanographic Data network among the Data Centers start growing up. And since Web technologies were being developed rapidly, became apparent that the above network had to be web based. From Centralized to Distributed model PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

The main obstacles that came through for this idea to be fulfilled were: Data bases was in different Schemas and Technologies specific per Data center so there was not any common data interchange mechanism. Data were not homogenized among Data centers. Data quality and Data vocabulary differed from Database to Database Main obstacles… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

From Non Spatial to Spatial Enabled Databases… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial • At the same time another obstacle was that due to premature Data Base technology many RDMBS systems lacked spatial capabilities. • So it was very difficult for the end user: • To query the Database with spatial criteria • To represent geographically the spatial information and produce geospatial products • To work interactive with Geo Data through interfaces with geospatial capabilities • To combine geospatial data in synthetic data products • Hence, for a distributed Oceanographic Data network to be established, the necessity of GIS enabled Databases became apparent as well. PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

From Non Spatial to Spatial Enabled Databases… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial • At the end of 90’s most of the common used RDBMS enriched with spatial capabilities and became then powerful mechanisms of integration, exploitation and presentation of spatial information • Worldwide oceanographic data centers, exploit this advantage and started upgrading their databases PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

Interoperability obstacle is remaining… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial • But still the luck of interoperability was unsolved. The spatial databases made researchers’ life easier with regard to querying data or representing graphically geospatial information.But… • they were acting without any interconnection among the various Data Centers. PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

Summarizing the historical brief throughout important E.U. projects PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial MEDATLAS 1994 1998 2002 SEA DATA NET 2006 MEDAR/MEDATLAS II SEASEARCH PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. • Efforts in QC • Data homogenization • Centralized management • Distribution with CD ROMs • Enrich existing data sets • Establishment of common protocols in QC and formatting • Enhance communication facilities (www, ftp, e-mail etc) • Development of a pan-european network for ocean and marine data & information management. • Improve the exchange, availability and accessibility of ocean & marine data and information. • Expand the online metadatabases with input of research institutes. • Improve the online accessibility of Sea-Search services & products. • Develop a standardized distributed systemfor managing and disseminating the large and diverse data sets • Enhance the currently existing infrastructures with web services FUTURE The new architecture An example Conclusions

At the same time with the efforts of the Oceanographic community, during the last five years new technologies aroused and became common known as: Web Services Web Services… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

First of all what is a Web Service? The hidden idea behind WS is the existence of a universal messaging mechanism, capable to deliver and translate messages for diverse and different systems. This feature came to open the road for the solution of interoperability. If different and diverse systems can interpret and process requests from each other let them stay different. Web Services fundamentals PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

There are various theoretical definitions of What Web Service is such as: “A software system designed to support interoperable machine-to-machine interaction over a network. It has an interface described in a machine - processable format (specifically WSDL). Other systems interact with the Web service in a manner prescribed by its description using SOAP messages, typically conveyed using HTTP with XML serialization in conjunction with other Web-related standards.” (W3C, 2004) What is a Web Service? PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

or “Web services are a distributed computing architecture. Only this particular architecture makes use of loosely coupled applications, as opposed to tightly coupled applications, to enable applications to communicate. This tightly coupled concept radically affects how information systems will work in the future” (Clabby 2003) What is a web Service? PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

Generalizing we may define that: “Web services provide a convenient and standardised way of exposing business logic over a network (the Internet) by use of specific components of communication” The basic components of WS are SOAP (Simple Object Access Protocol) which provides a Standardized messaging structure based on XML UDDI (Universal Description, Discovery and Integration) which provides the ability to dynamically select a service at runtime WSDL (Web Services Description Language) which describes how the client communicates with a Web service or what a service does or where is located. What is a web Service? PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

In the most simple form a web service architecture may be depicted as the diagram Web Service Architecture… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

The rapid development of technologies towards WS concept has produced several sets of Web Services with regard to the area of the WS applicability. Especially for Geospatial data, Open Geographical Consortium (OGC) has defined two basic sets of Geospatial Web Services. Geospatial Web Services PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

The Web Map Services (WMS) which are concerned with transforming spatial data into maps (images). The Web Feature Services (WFS) which are concerned with direct access to data - reading, writing, and updating data. WMS and WFS… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

Furthermore, more Web services are under finalization process, one of the most important of them is WPS (Web Processing Service) WPS is designed to standardize the way that GIS calculations are made available to the Internet. Can describe any calculation (i.e. process) including all of its inputs and outputs, and trigger its execution as a Web Service. Supports simultaneous exposure of processes via GET, POST, and SOAP, thus allowing the client to choose the most appropriate interface mechanism. Web Processing Services PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

The HNODC data management case. HNODC… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

Hellenic National Oceanographic Data Center (HNODC) is a National Public Oceanographic Data provider and at the same time a member of the International Net of Oceanographic Data Centers (IOC/IODE) HNODC owns a very big volume of Data and Relevant information about the Marine Ecosystem HNODC… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

For the efficient management and exploitation of these Data, the first step was a relational Data Base to be constructed. After the necessary stage of data quality control took place, following the International standards, as they agreed during several EU Projects or International Organizations. Finally a mass volume of over 300.000 station data concerning , physical, chemical and biological Oceanographic information, stored in a Relational Data Base. HNODC… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

The next target aroused then, was an Application to be constructed capable to represent either - the Geospatial aspects of this information - together with the non spatial information The necessary analysis took place and a traditional Web Application was developed by the use of the convenient web tools and technologies. Geographical representation was achieved by the use of a Middle Layer tool (ArcIMS) over the Data Base Mechanism. HNODC… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

The convenient Multilayer Architecture was followed The Relational data base as the back end. ArcIMS as the Geospatial representation Mechanism. Apache and Tomcat in heart of the Middle-Tier System. On the top a web Interface with selection criteria for querying specific data sets in specific geographical areas. HNODC - Architecture… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

HNODC - Architecture… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

HNODC – Web page… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

Although it’s geospatial capabilities this Application stranded only as a data geographical presentation and queering mechanism without any Data management and Data processing abilities. and also without any Data interchanging with other applications and with it’s own data vocabulary HNODC… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

At the same time technical issues aroused, regarding the Application’s performance, due to the inefficiency of the Geographical mechanism to handle big volumes of data sets as fast as a Web Application needs. For HNODC this approach is today considered as the fist Version (V1) of an Integrated Web Based Environment for it’s Data Management and Data exploitation. HNODC… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

Thus and at the time that Web Services became a fact, a second Version was planned The target of this effort was an Integrated Web Based environment to be developed, following at least the basic concepts of a Service Oriented Application (SOA) which are: The interface contract to any WebService is platform-independent. Any Web Service can be dynamically located and invoked. Any WebService invoked is self-contained. That is, the service maintains its own state. HNODC… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

For the development, state of the art software components and tools replaced the “old” ones: Geospatial and no Spatial Web Services mechanisms took the place in the Middle- Layer. Geospatial open source tools and custom development was employed in presentation layer. HNODC… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

At the same time the application may now interact with any other SOA application either in sending or receiving Geospatial Data, since it inherits the big advantage of interoperability between Web Services systems. HNODC… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

The Architecture can denoted as follows: At the back End Open source PostgreSQL DBMS (and not only..) stands as the data storage mechanism with more than one Data Base Schemas cause of the separation of the Geospatial Data and the non Geospatial Data. UMN Map Server and Geoserver are the mechanisms for Represent Geospatial Data via Web Map Service (WMS) Querying and Navigating in Geospatial and Meta Data Information via Web Feature Service (WFS) And in the near future Transacting and processing new or existing Geospatial Data via Web Processing Service (WPS) HNODC - Architecture… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

WhereGroup MapBender, a geospatial portal site management software for OGC and OWS architectures, acts as the integration module between the Geospatial Mechanisms. MapBender comes with an embedded data model capable to manage interfaces for displaying, navigating and querying OGC compliant web map and feature services (WMS and transactional WFS). HNODC - Architecture… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

Apache and Tomcat stand again as the Web Service middle Layers Apache Axis2 with it’s embedded implementation of the SOAP protocol acts as the No spatial data Mechanism of Web Services. (These modules of the platform are still under development but their implementation will be fulfilled in the near future.) Finnaly Web user Interface for the end user developed based on enhanced and customized version of a MapBender GUI. HNODC - Architecture… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

HNODC - Architecture… PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

HNODC Web Interface PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

Conclusions PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial • As conclusion we may denote that embedding these new technologies to data management model we can exploit the advantages to: • Inherit interoperability between among Data Centers and Data Providers • Interchange and combine Data Sets with minimum programming efforts • Produce comprehensive and self explained new Data Products combing various and diverse data sources • Help end user to navigate through a wide range of information PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

We are happy to say that HNODC joins the global scientific community by providing and consuming application Independent data products. Conclusions PAST Historical Brief Centralized D.M. From Centralized to Distributed Model From Non Spatial to Spatial PRESENT Web Services W.S. Architecture Geospatial W.S. WMS/WFS WPS HNODC General HNODC Architecture HNODC Interface HNODC W.S. FUTURE The new architecture An example Conclusions

The quality control procedures that HNODC uses to manage oceanographic data and information follow the international standards recommended by IOC, ICES and EU: Documentation of the data sets Format Conversions (Medatlas) Quality Control on data and metadata Quality Control in HNODC

Quality Control at Metadata and Data (automatic and visual): Format, codes and completeness of information Cruise and stations headers Observed data of vertical profiles and time series of fixed moorings The results of the QC are added as quality flags to each numerical value for the location, date, bottom depth and the data points of vertical profiles and time series. The quality controlled data set is archived in the HNODC’s multidisciplinary database. Quality Control in HNODC

The main software tool deployed for QC is “Scoop” Scoop is a open source application aimed in moderating and visual handling of Quality Control aspects like : Checks in various data format Duplication entries in datasets Visual observation of vertical or time series profiles …etc Quality Control in HNODC

Format Checks

Automatic Checks of Cruise and Stations headers Vertical Profiles • duplicates cruises within a preset space-time radius • date of the stations • ship velocity between two consecutive stations • bottom sounding (ETOPO5) • land position (GEBCO) Time Series • duplicates entries • Sensor depth check • Series duration check • land position (GEBCO)

Observed data of vertical profiles and time series of fixed moorings Automatic Checks of Observed Data • at least two parameters required for the qc (vertical reference + measurement) • constant profiles or times series (sensor stuck) • data out of regional values • comparison with pre-existing climatological statistics (LEVITUS, MODB, MEDATLAS). Time series are compared with internal statistics. • decreasing reference parameter (pressure or time) • pressure must not exceed the bottom depth • spikes • vertical instability (for vertical profiles)

Visual Checks of Cruise and Stations headers

Visual Checks of Observed Data

Implementation through European Community Research Projects