Second Generation Spatial Information Warehouse Architecture

1. Henry Kucera HydraSpace Solutions Inc www.hydraspace.com Second Generation Spatial Information WarehouseArchitecture

2. HydraSpace Development Partners • System Integrator • Henry Kucera, • President, HydraSpace • Development Teams • Edric Keighan, • President, CubeWerx • Pierre Lafond • President, Holonics • Gail Kucera • President, Mercator

3. Strategic Goals • The strategic goal is to: • achieve greater interoperability by establishing a common information infrastructure for use by all stakeholders • promote and leverage existing and evolving Technology and expertise (e.g., AGMIID, OGC Web Mapping, CEONet, MID-C, AquaGIS, III/OGDI, and others); • gain competitive advantage by consolidating various independent initiatives; • provide entry to foreign markets by cooperating with similar international activities; • ensure the compliance of products by funding standards-based activities to influence international standards evolution; • demonstrate a solution for the global delivery of integrated geospatial information, applications and services.

4. Information Components Framework information Stakeholder Information Endangered species Siesmic surveys Facilities information Marine habitat Geological mapping Climatology Shipping and aeronautical Topographic base Survey and GPS Hydrographic Charts Land disposition (leases, tenures, water licences, etc.) Administrative areas Satellite imagery Tide and Current …

5. Inconsistent data formats • Proprietary software & hardware systems • Varying spatial & temporal resolutions • Variable accuracy & precision • Multiple views of common entities Connectivity Barriers:

6. Pesticide Permit Mine Operations Environmental Assessment Act Forest Tenures X Watersheds Management Waste Permit Water Licenses X X Lakes Database Fisheries Stream class A Ecosystem Integrity X Fish Stock Releases Rare & Endangered Species Hatchery Mgt X X X Petroleum Exploration (biophysical habitat WETLAND) Fisheries Coastal Zone Management Shipping of products Modified from BC Ministry of Environment (1997) The Enterprise Information Model ….For supporting regulation and governance

7. Production testing Export Applications Environmental Assessment Engineering Historical Research Reservoir Testing Refining Marketing Data Synthesis Providing Delivery Infrastructure Production Exploration Drilling Seismic Profiling Land & Permit Acquisition Partnerships Consumer The Enterprise Information Model for Supporting Industry… Same data - different view

8. Distributed Spatial Architecture Requirements The data handling requirements are: • a data warehouse, conforming to international standards, for the storage and integration of spatial and non-spatial information • a distributed architecture for locating and accessing data through a simple single-window search • a suite of tools and services for data load, integration and retrieval, Web browsing and report generation • The system must be designed to deliver information in a form readily accessible and useable by decision-makers in support of economic,environmental and environmental obligations.

9. Distributed Spatial System Functions The system must support the following: • integrated spatial and non-spatial data capture and storage • capability to catalog data warehouse contents • linkages to multiple warehouses • timely location, access and retrieval of operational and warehouse data • multi-resolution views of data • reporting, display and visualization of raw, processed and abstracted data

10. Enterprise support for Multiple Tools • Data Access Tools • formulate requests • extract geodata • visualize geodata • navigate in the data warehouse • Analysis Tools • spatial analysis • capture, compilation, edit • any long transaction • Data Mining Tools • application specific

11. GIS & applications (ESRI, CARIS, INGR) Integration Tools Tools for modeling, cleaning, integrating and loading data. Access Tools Tools for query, analysis and reporting. (Web-based preferred) Tier 3 Application Environments Tier 2 Application Servers DB servers (CubeSERV) File servers (SAFE FME) App Servers (CIDAS) Data Access Protocols & APIs Network Interface APIs - (OGDI, OGC & CGI) File manager Meta-Data Management (Repository) Tier 1 Data Management & Data Server Environment Data files Data Management Tools Object-Relational DBMS + with Spatial enhancements Multi-tier Spatial Architecture Components

12. Medium & Thick OGDI Clients Thin Client Medium & Thick SQL Clients TIER1: Applications Web Server TIER 2: Application Server Components OGDI Server CubeSERV Server CubeSTOR Server SQL Engine Text Cartridge CubeSTOR Spatial Cartridge SQL std. Cartridge Other Cartridges TIER 3: Oracle Data Server Oracle Network Computing Architecture File manager File manager Data files File manager Data files Data files • Spatial data • Attribute data • Meta data Database Files Mapping Multi-tier to the Spatial Data Warehouse Architecture

13. Picture Case (Thin Client) Display Graphic Element Case (Medium Client - OGDI & SAFE FME with GIS browsers) Render Service Data Case (Thick Client - SQL Web Applications in conjunction with either of the above) DEG* Service * DEG =Display Element Generator † SF = Simple Features Filter (SF† Query) Service OGC Service Model for Levels of Distributed Data Access

14. Information Access:The Picture Case • OGC Compliant CubeVIEW using Common Gateway Interface (CGI) The “Picture” case Using a CGI-encoded set of instructions, a portrayal can be constructed into a GIF or JPEG. That GIF or JPEG travels to the browser via HTTP.

15. Information Access:The Graphic Element Case • Open Geospatial Datastore Interface (OGDI) The “Graphic Element” case A Java applet or ActiveX component connects to a service (in this case using GLTP but in the future it will use XML/HTTP, IIOP, etc.) and requests either Display Elements (raster vector) or Rendered Images (e.g. transparent GIFs). These pieces can be rendered or layered on the client side. Currently MapObjects, FME ArcView, GeoMedia, PCI and other applications can access data using the OGDI.

16. GIS or file archive Spatial Browser Custom application Vanilla HTML Stakeholder Data FME Translator FME with OGDI OGDI Server SQL Html/ CGI OGDI Reader Multi- dimensional Data Format (MDF) CubeSTOR Convert Framework Data CubeSTOR Load • Spatial data • Attribute data • Meta content Information Access:The Data Case The “Data” case There is interaction with online sources via links and client initiated service connections. The data files might contain several feature sets, tables and some symbolization information.

17. Distributed Warehouse: Access to data at any level from anywhere DATA ACQUISITION Picture, Graphic elements or Data OPERATIONAL & EXTERNAL DATABASES LOCAL DATA WAREHOUSE MASTER DATA WAREHOUSE LOCAL DATA WAREHOUSE LOCAL DATA WAREHOUSE WEB PLUGINS & HTML/JAVA TOOLS & APPLICATIONS DATA VIEWS Open Distributed Spatial Environment Http://www.mercator-sys.com

18. ISO SQL3 & SQL/MM Common Language Common Data Model Common Spatial Functions Repository, Dictionary & ADT Support APIs (OGDI/CLI/ RDA) Application Level: User-defined Repository Design Physical View Level: Level: Level: Database Subject Area Information Model Model Schema ISO TC211 OGIS Common Meta -Model COM/CORBA Compliance Spatial Features & Operators Simple Feature SQL Modelling Methods JAVA & OLE/DB Meta -Information Web Mapping Standards Services Definition Protecting your information!Warehouse is based on International Standards:

19. Warehouse products: • CubeSTOR Convert • CubeSTOR Spatial Cartridge for Oracle 7 and 8 • CubeSTOR Server • CubeSTOR Load • CubeSTOR Archive • CubeSERV (Web based product) • CubeVIEW (Web based product) • GenSystem (Network and Generalization tools) • SAFE FME/OGDI (& FME Direct) • Compusult CIDAS and MetaManager • Geomedia/OGDI • MapObject/OGDI and ArcView/OGDI (ArcInfo/OGDI in progress) • PCI/OGDI • …. others

20. CubeSTOR Access... • Existing CubeSTOR/Oracle Database Interface (release 2.09) • CubeSTOR Dictionary Views • ODI Packages • Seamless storage of all types of vector data • seamless storage of multi-channel imagery • Storage that supports hierarchical cellular data structures • Query through OGC compliant web mapping servers • Support for storage and return of OGC/SQLMM WKB • Support for query through OGDI • Integrated CGI interface • 3-D points, lines and polygons • Open API to multiple tools

21. CubeSTOR Access... • Additional function in the Next Release CubeSTOR/Oracle Database Interface ( SQL3 Oracle 8 features for release Dec. 1999) • Feature partitioning managed by SQL engine • Insert, Update, Delete operations using standard SQL • Base Functions on Well-Known-Types • Metric Functions on Well-Known-Types • Geometric Manipulation Functions on Well-Known-Types • Support for ADT’s • Topological Operators (4 intersection method: Egenhofer and Clementi)

22. Installed CubeSTOR Warehouse Systems • Canadian Forest Service - National Forest Information System (60 GB aiming at 350GB in the next 4 months and expansion capability to 1TB +. The immediate expansion is to load the BC Forest Inventory spatial data (FC1 + FIP) as well as yet to be determined work with Ontario Ministry of Natural Resources. • Department of National Defense - Eight installations to manage and map topographic and intelligence. (100 GB per installation) • US Army Topographic Engineering Center - Multiscale information management. (20 GB with growth to 100 GB in 2000) • Health Canada - Spatially enabled health statistics. Digital Chart of the world and National Topographic Database plus other datasets. (Estimated size 50 - 60 GB) • NRCan Build of an image library (underway). (12 GB - may grow to terabytes) • NRCan Host of Digital Chart of the World (20 GB) • Canadian Hydrographic Service - Manage soundings and chart data at five locations (Terabytes)

23. Input Data Integration SLF ITD 1:50K PITD 1:250K • Imagery • LIDAR • SAR • etc Tactical Data Vector Data VPF VMAP 1 & 2 VITD 1:50K UVMAP 1:10K CADRG 1:250K Business Data Usage INFORMATION GENERATION COMMAND & CONTROL REQUIREMENTS & TRAINING POLICY & PLANNING Data Assimilation & Management INTEGRATION & CONFLATION SERVICES DATA MANAGEMENT & PRODUCT DISSEMINATION DISTRIBUTED PRODUCTION Case Study 1: US ARMY TEC- AGMIID

24. SLF VPF Existing interface VITD 1:50K ITD 1:50K PITD 1:250K VMAP1 & 2 CADRG 1:250K UVMAP 1:10K Enhanced interface Year 2 interface OGDI Enhanced FME Retrieve features w/ IDs CubeSTOR/Oracle Database System Features with meta-content Linked Features with Meta-content Information “Products” with meta-content OGDI GEODEX Visualize feature data Feature Linking Service (FLS) Conflation/ Generalization & Best Path Links in MSAccess Year 1: US ARMY TEC

25. DYNAMIC CONFLATION + + Imagery Vector Data Elevation Data Complex query New “Product” VALIDATION & VERIFICATION Linked Query w/ Boolean CubeSTOR/Oracle Database System with FME/OGDI/OGC Interface 4 Interface 1: Update data and meta Clean Source and/or Features Linked Features with Meta-content Conflated Persistent Information “Products” 1 2 3 DATA MINING & PLANNING Complex Query Interface 2: Plus new data Interface 4 Interface 3 SIMPLE QUERY Conflation/ Generalization Analysis and linking Year 2 & 3: US ARMY TEC

26. Multiple datasets Medium & Thick OGDI Clients Thin Client Medium & Thick SQL Clients TIER1: Applications Web Server TIER 2: Application Server Components Spatial Load Facility (FME & OGDI) Data Cleansing , Topology Validation & Quality Assurance OGDI Server CubeSERV Server CubeSTOR Server Network Analysis SQL Engine Generalization GenSystem Text Cartridge CubeSTOR Spatial Cartridge SQL std. Cartridge Other Cartridges Engine TIER 3: Oracle Data Server Oracle Network Computing Architecture • Spatial data • Attribute data • Meta data Database Files US TEC-AGMIID Target Architecture:Dynamic Multi-scale Visualization and Analysis

27. Topology integration Version integration Visualization & reporting Metacontent maintenance Schema transform Load & QA Best path Feature extraction Generalization Spatial Warehouse Data Server Targetted Services to be delivered to TEC through AGMIID

28. Case 2: NFIS - CGDI “The National Forest Information System (NFIS) is a policy and science framework designed to support Canada’s national and international commitments by providing timely and accurate spatial and non-spatial forest resources information”

29. NFIS: Project Deliverables • The first phase of the project completed in March 1999 with the following deliverables: • Design and implementation of a working multi-gigabyte spatial data warehouse based on an Oracle/Unix platform • Population of the warehouse with selected metadata and data sets including: CANFI, NatGRID 1km & 100m elevation data, classified satellite imagery, protected areas, selected provincial forest inventory information, climate change data • Criteria & Indicators Reference to 1km NatGRID • Incorporate a Web-based reporting tool for National Forest Inventory

30. NFIS:Reporting and Analysis Services

31. Spatial Load Interface ORACLE 8 Environment Open GeoData Interface (OGDI) SQL Language Interface Output Server GIS CubeWerx Spatial Cartridge Spatial - ESRI EOO - Bentley DGN - Satellite - DEM - SAIF Feature Manipulation Engine (FME) Abstract Data Types (OGIS and SQLMM Compliant) OGDI Server Driver & CGI/BIN API 4GL Apps Open Geodata Interface (OGDI) Oracle Database Kernel Common Gateway Interface (CGI, HTML JAVA) Meta-content Schema Non - Spatial - Oracle - DB2 - flat files Web Apps Non-spatial Schema Spatial Schema Seamless, pluralistic Standards-Based Diverse Data Formats Common Application Server Interfaces Client Applications Multi-media Database Server The technology environment for forestry reporting

32. NFIS: Criteria and Indicators Pilot C&I Test Data Forest Inventories Protected Areas Ecosections Forest Regions Biogeoclimatic Watershed Atlas Species Status (CDCs) Allowable Cut Data NatGRID C & I Spatial and non-spatial data data updates NFIS Data Warehouse NFIS Catalog of Available Data Reporting Requirements Other Data Warehouses Gap Analysis C&I CCFM Reporting 1.1.1; 1.1.2; 1.1.3; 2.2.2; 2.3.1; 4.4.4; 4.5.1; 5.1.2; 6.5.1 C & I Analysis and Synthesis NFIS Data Access Methods

33. Biogeoclimatic (Provincial) Forest Cover 1:20000 (Local) Terrestrial Ecozones (National) Spatial and Attribute Data Warehouse NFIS Data Sampling Every data set will be sampled at a one second interval

34. Sampling points HHCODED 1 second grids on the forest cover.

35. Data Sampling Sampling at every second by every second grid. Every sample point will have a unique HHCODE along with the coordinates, and attributes from the source data. A 1:20,000 mapsheet is 6 minutes by 12 minutes 6 minutes x 60 seconds = 360 seconds 12 minutes x 60 seconds = 720 seconds 360 x 720 = 259,200 samples per mapsheet. 100 6’by12’ mapsheets per letter block = 25,920,000 points Study area, 3 letter blocks = 77,760,000 points

36. Data Loads A 1:250,000 is (100 1:20,000 ) 1 degree by 2 degree 3600 x 7200 = 25,920,000 There are approximately 7000 1:20,000 mapsheets in B.C. 259200 x 7000 = 1,814,400,000 1,814,400,000 records for British Columbia approx. 20,000,000,000 records for Canada

37. Enable Analysis & Decision Support • Handle Large Volume of Spatial Data, • Leverage existing technology, • Use Basic Tools to keep training and acquisition costs low Case 3: National Health Surveillance System (NHSS) • Link NHSS GIS Prototypes

38. Any GIS client CubeView Formida Applications (next phase) client Presentation Layer Distributed Web interface Page (OGC) Nation Health Surveillance System application server Application Server Layer (Currently in implementation Data Server Layer OGDI Interface and Drivers foundation spatial data foundation spatial data Proprietary GIS Vital statistics Air quality data format & vital CSD & detailed attribute - SO Aggregate data 2 Detailed data stats & socio- - Battery flares economic (Alberta Only) CSD & EA CSD, EA & On Client/ Tunney's Pasture CubeWerx Hull Socio-Economic Replicated WIN/NT Win/95 CubeWerx Hull ODBC interface Tunney's Pasture Grasslands MID/MIF and DBF UNIX UNIX WIN/NT OGDI interface OGDI interface OGDI interface MID/MIF MID/MIF MID/MIF

39. Case 4: CHS Source DBFrom paper charts to digital data From • Map sheet paradigm • Closed, single purpose, file based GIS applications • Single purpose internal “cartographic” systems • Standards for data exchange • Moderate data volume To • Continuous coverage • Open, object-relational DBMS with many end-user tools • Multi-purpose “decision support” systems • Standards for data access • Massive data volume

40. RealTime Data Metacontent Clean Data Data Products “Refined” Source Data Source Data CHS Back Office Architecture & Flow

41. Application Access Federated Data Warehouse Interface Layer • Open Interface • Flexible Architecture • Expandable • Neutral • Rapid Access • Easy GIS Tools RealTime Data Metacontent Data Products Visualization Tools Web Browser “Refined” Source Data CHS Front Office: Open Access to clients

42. Metacontent Management through the Business Life cycle Prospective Metacontent CUSTOMER Marketing Distribution Planning Production Product Data Management External collection Survey/ internal collection Data Acquisition Static and Dynamic Metacontent

43. Halifax St John’s Montreal Ottawa Distributed process architecture • Remote collaboration handled through standard distributed database architecture and networking Survey hydrographer Validation hydrographer Cartographer Production Quality Control

44. CASE 5: GEOIDE Feature Extraction and GeneralizationLinking industry and academia “develop a service to extract information from high resolution imagery using collaborative data and place it into an open distributed data architecture that can be used for such data analysis and warehousing capabilities as feature linking, generalization tools, aggregation, and visualization.”

45. Goals for Feature Extraction Images Multi-resolution Multispectral Multisensor User requirements / goals • Semi-automated Feature extraction • Line features • roads (CCRS) • rivers (UVIC) • coastlines (JOINT) • Area Features • lakes • islands • forest stands • agricultural areas etc Operator

46. Simple stream extraction algorithms complete • The project is using commercially available imagery such as 4m KFA 100, 1m IKONOS (when available), and 1m digital orthophotos. • UVIC grad students have also developed remote sensing tools to capture necessary information.

47. Medium & Thick OGDI Clients Thin Client Medium & Thick SQL Clients Multiple datasets Web Server Spatial Load Facility (FME & OGDI) Data Cleansing & Topology Validation & Quality Assurance OGDI Server CubeSERV Server GeoMedia CubeSTOR Server Network Analysis SQL Engine Feature Linking Service (FLS) Alpha Feature Extraction Service (UVIC-CCRS) Generalization Text Cartridge CubeSTOR Spatial Cartridge GenSystem SQL std. Cartridge Other Cartridges Engine Oracle Network Computing Architecture • Spatial data • Attribute data • Meta data Database Files Dynamic Multi-scale update, generalization, visualization and analysis • Seamless information • generalized products • automated data mining

48. Proposed Feature Extraction Deliverables • The first phase of Feature Extraction to be completed by October 30, 1999 with the following deliverable: • Design and implement a working (not integrated) feature extraction demonstration • The second phase deliverables in early 2000 would include: • interfaces from the Feature Extraction service to the spatial data warehouse based on the CubeSTOR/Oracle implementation on a Unix platform • Population of the warehouse with selected metadata and data sets from potential partners including: LIDAR, SAR, VMAPX, VITD, PITD, ITD, classified satellite imagery, selected tactical information. • The third phase in late 2000 - early 2001 would: • Develop and demonstrate the data fusion enhancements for terrain analysis and extraction of higher-dimensional features

49. Benefits of adding feature extraction to AGMIID The enhancements would result in: • a feature extraction service for multi-resolution imagery • images, hi-res vector, and classified output in the data warehouse, accessible to other services (e.g., feature linking and terrain analysis • a distributed architecture for acquisition and update of data through a single portal • a suite of tools for data load, integration and retrieval, Web browsing, and report generation • The system must be designed to deliver information in a form readily accessible and useable by decision-makers.