WTF-GOFC Technical Team Meeting

1. WTF-GOFC Technical Team Meeting Allan Doyle NASA/SGT November 27 - December 1, 2000 Greenbelt, MD, USA

2. Agenda -Monday 0900-1015 Intro & Overview Tech Team Goals - Allan GOFC Datasets - Matt WGISS Menu - Bernhard OGC Descriptions - Allan 1 • 0830 • Breakfast • 1030-1230 Demo Deconstruction • Looking for interface points and reusable chunks… • 20 minutes each, maximum - don't follow a script, • show technical ideas and issues • - TRFIC • - MOCHA • - ESRIN Fire Atlas • - JRC FireWeb • - CCRS GOFC demo • - NASDA JERS data access • 0900 • Intro/Overview 3 • 1030 • Demos done 1230 Lunch • 1330 • Current status • 1330-1445 Current status - Everyone • What "works" now, what doesn't? • What are areas that need improvement? 2a • 1500 • Science View • 1700 • IP Issues • 1500-1645 Science View - Dave, Chris, others • Science discussion with GOFC scientists • Science priorities -- what is GOFC trying to do in 2001? • What should the WTF-GOFC focus be? 2 1730 Adjourn 1800 No-Host Dinner ? • 1700-1730 IP Issues - Dave • Intellectual Property - how do we provide an open atmosphere?

3. 0830 • Breakfast • 0830 • Breakfast • 0830 • Breakfast • 0830 • Breakfast • 0900 • Wrap up 1200 Adjourn 1230 Lunch 1230 Lunch 1230 Lunch 1730 Adjourn 1730 Adjourn 1730 Adjourn Agenda - Tuesday through Friday • Items • Monday "overflow" • Demos • Science discussion • OGC WMT • Science Priorities & Needs • Catalog/Data Interfaces • Core set of what can be done for next year • Agency implementation plans for next year • GOFC Datasets • WTF-GOFC Architecture • WGISS Menu • OGC EO SIG next week • Future Plans • Others as generated…

4. Our charter • How can WGISS capabilities be pulled together to support a project like GOFC • Evolution of GOFC • Evolution of WTF • Demos - serve two purposes • Focus work onto a goal • Are opportunities to get people on board • Provide a common vocabulary & definition of interoperability

5. Intro & Overview

6. Desired Outcomes • WTF-GOFC Technical Team • What's our mission? • What's "in scope", what's "out of scope"? • Milestones • Work plan • 1 year plan for WTF-GOFC • Demo plans (Kyoto, others?) • Technical goals • Other? • 5 year plan for WTF-GOFC • What's the "ultimate" goal?

7. The cycle is completed as these Products and Services get taken up by WGISS into the WGISS Menu. A Proposed WGISS Test Environment Version Sept 12, 2000 WGISS Data & Information Products 1 • A Nakodo process will allow the creation of a WGISS Test Facility for the projects. • An improved set of Products and Services should be one result of this process. • Projects and the WTF also have links to the “outside”. E.g. WTF can be deployed for long term operation. • The Test Facility can use Products and Services from the WGISS Menu and from the Projects. And can feed back to the Projects! • These are being collected into a “WGISS Menu” which will offer a single place to find them. • There is a set of tools and services (e.g. IDN, DIAL) • A partnership of programs/projects will be able to draw from this menu Agglomeration oftools and servicesheld and operated by CEOSorganisations 2 3 WGISS Menu 4 Improved Products & Services Products & services Open reference list of WGISS products & services, updated on the basis of the results of the test facility, linkedon-line to the products and servicesthemselves 5 Based on the test facilityresults 6 7 Links to data suppliers Products & services 8 Disaster Management Disaster Management 9 Oceans Oceans GOFC GOFC Partnership WGISS Test Facility Projects Selected WGISS products made into a coherent, open, modular system by the partners to address each project’s needs (can include tools dev’d by the project), dev’d & tested against the project’srequirements Nakodo Define the data & information systems and services requirements Tools, data Products& services Non-WGISS requirements Long term operation A Proposed WGISS Test EnvironmentVersion Sept 12, 2000

8. DISCOVERY ACCESS VISUALIZATION ANALYSIS WORKFLOW Combination Mensuration Portrayal Directory Subscription Query/Filter Ordering Notification Reprojection & Coord. Trans. OnlineAccess Overlaying FormatConversion Inventory Publishing Subsetting/Subsampling AlgorithmicTransformation Mosaicking Scripting/Chaining Statistics Classification Drill-down Animation Featuresw/GeometryData Server Level 1Data Server GriddedCoverageData Server Candidate Core Data Services

9. CEOS/WGISS Components("WGISS Menu") Directory (via IDN), Inventory, Ordering, Direct access Directory, Inventory, Ordering, Subscription Directory, Inventory, Ordering, Direct access Directory, Direct access Directory, Inventory, Ordering, Direct access, Format Conversion, Subsetting, Overlaying, Animation, Portrayal Portrayal, Overlaying Combination, Algorithmic Transform, Mosaicking Reprojection & Coordinate Transformation Infrastructure Services Infrastructure Services

10. WTF/GOFC activity • Interact with science teams and WGISS subgroups • Jointly define targets • Develop bottom-up iterative collaborative prototyping process • Analyze demos that have been shown • Identify services they provide • Identify services they could re-use from WGISS Menu • Identify potential points of interoperability • Develop architectural view of services • Support prototype development and next demonstration series • Based on science targets and technical architecture • Spin off WTF prototypes to regional networks for operational use

11. Data Services Data Services Data Services Science Community Views WGISS Products & Services WGISS Test FacilityTechnical Vision Using the WGISS Products and Services, new applications can be built while preserving a project or community feel.

12. Introduction to Interoperability • The next few slides are meant to illustrate what is meant by interoperability and how it can be modelled.

13. Heats food Must heat to at least 160 deg. F Provides & contains water Must hold at least 12 liters Keeps things cool Must be able to keep below 32 deg. F Abstract Model

14. 4 burners 30” wide 220V, 60Hz 4 rings 80cm wide 220V, 50Hz 36” wide 3/4” hole for faucet 100cm wide 2cm hole for tap 30” wide 110V, 60Hz 80cm wide 220V, 50Hz Implementation Specs

15. Essential Model Kitchen Floor Counters Sinks, stoves, and refrigerators can be arranged in several ways in a kitchen. If arranged to form a triangle, the kitchen can be used more efficiently than if arranged in a line

16. Relationship between models • Dependency between EM and AM • Which comes first? • Can one exist without the other? • Dependency between AM and IS • AM should* come first • AM can be quite high level and can be refined as IS instances provide feedback • * or, it should look like it came first.

17. (Abstract Model) (Implementation Specs) DCP #1 f f a a b b DCP #2 A simple example b = f(a) How does f( ) get a? What “travels” through the = sign? Realization or footprint Serialization

18. Good AM = Easy f f a a b Realization or footprint Serialization b Bad AM = Hard Interoperability comes from the AM • How do you ensure that the realization and the serialization are easily mapped to/translated? • You have to have something in the AM that helps do it or interoperability suffers

19. Types & Behaviors(or Data and Operators) Directories Metadata Service Registries Catalogs Public Behaviors Public DataTypes Metadata Metadata Private Behaviors Private DataTypes Footprint/Serialization Issues Algorithmic/Functional Issues “Data” “Operators”

20. GOFC Datasets • Matt Schwaller's slide(s) go here… Discussion: • What services exist today to support the datasets? • Do we need finer grained views into the datasets?

21. WGISS Menu • Bernhard Buckl's slide(s) go here… Discussion: • WTF-GOFC TT should build a "WTF-GOFC Menu" • On the WGISS menu, the views were very generic. Better to design a web site to support particular functionality for the community • 3 views, possibly multiple instances • User Community View(s) • Project View(s) • Technical View

22. Science Discussion • Chris Justice - insert slides here… • Dave Skole - insert slides here…

23. Applications (viewing, editing, discovery, analysis, publishing) Catalogs & Directories Operators & Models Repositories http://ceos.org 100,50,2 300,75.4 X=56 OGC Interoperability Program(Core Architecture) Network Services & Security / E-Commerce

24. Applications (viewing, editing, discovery, analysis, publishing)  Dynamic Portal  LO Clients  Viewer Clients  Discovery Clients Catalogs & Directories Operators & Models  Geoparse, Geocode, Geolink  Service Registry  Coordinate Transformation  Catalog  Web Map Server Repositories  Web Feature Server, LO Folder & GML http://ceos.org 100,50,2 300,75.4 X=56  Web Coverage Server & IML OGC Interoperability ProgramCurrent Efforts in Web Mapping Testbed & Geospatial Fusion Services Testbed Network Services & Security / E-Commerce  General Service Model & Security Demo

25. TRFIC Demo • Schematic of the TRFIC system and 3 cases of access • Broker model • Allows TRFIC client to expand its available data resources • Polygon upload - currently shapefile, moving to GML. • Client side layering • What's hard (in order of near term to long term issues) • Wire together the different warehouses to any "brokerage". Peer with EDC - FGDC metadata and browse products. • How do you find out all the owners of a particular Landsat scene • Warehouse/Broker/User, Peer-to-Peer, Co-operative • Decentralized production of derived products (automated, distributed) • Decision support systems • Impediments • Tendency to not allow for distributed approach • Network challenge • Bandwidth to tie the servers together. Clients don't need much bandwidth.

26. Science Views Chris Justice - GOFC Fire - See separate presentation Dave Skole - GOFC Forest Cover - See separate presentation

27. World Fire Web • Distributed processing at each receiving station • Collected by JRC • Products delivered via WMS 1.0 interface • Ultimately the products will be delivered direct from receiving station to end-user via WMS interface http://opengis.jrc.it • http://soda.jrc.it - advertising service under development Discussion: • Would be nice to get access to regional data directly

28. ESA Fire Applications • Implements a pipeline that uses OGC specs, but does not (yet) publish access points into that pipeline • Provides different levels of access • About to begin near-realtime access to European MODIS Level 1b data

29. Data set discussion • Identify key data sets/data holdings & their locations, what data goes with what services (or vice versa) • Issues that are not related to technology • Landsat 7 at $600/scene x 1000's of scenes is expensive • SPOT is commercial, more $/scene, reproduction limitations • Tracking issues - Ikonos has to be tracked as it leaves the warehouse (who, what use, etc.?) • Tracking requirement may not extend into derived products (to be researched) • Regional Mosaics are a good idea - WTF should provide composite mosaics for regions that are "pre-assembled".

30. Fire Data Discussion Notional "by Tokyo" scenario Source Lat/Lon Time Confidence MTSAT ATSR AVHRR DMSP MODIS TRMM Local GIS Data Brokers Users Landsat Browse

31. Tuesday … Chris Justice - more details - near term/further out - data bundles - what do we want to do? • Items • Monday "overflow" • Demos • Science discussion • OGC WMT • Science Priorities & Needs • Catalog/Data Interfaces • Core set of what can be done for next year • Agency implementation plans for next year • GOFC Datasets • WTF-GOFC Architecture • WGISS Menu • OGC EO SIG next week • Future Plans • Others as generated…

32. Tuesday

33. Data bundles • 3-4 bundle types, national and regional scale • Emissions bundle - satellite, in-situ, model output • Carbon bundle - hi res input data, derived products, rate of change, areal extent, biomass, forest inventory • Collected from several producers into a bundle • Broker might provide service such as transformation to uniform grid • Ordered based on spatial area and time • Put these together in one place or portal access point instead of having them be separate • Media or online based on users' connectivity • Most operational users want • high level products & subsets • Systematic deliveries of data on a regular basis • Some want raw data

34. Data types • Volume, projection, formats, services/operations • Services • Find data • Get data • Subset - band, region, time • Reproject/coordinate transformation • Visualize on local desktop, combine with local data

35. GOFC FIRE - notional, preliminary… • Near term - multi source data - 3-18 months • We can learn about • Initial data bundle • Initial cut at services • Initial architecture problems • Subscription and Notification Services • Should work with what people have right now • Should provide path to standards… Publish/ Discovery ATSR AVHRR DMSP MODIS TRMM Bundles Source Lat/Lon Time Confidence Local GIS Data Producers Brokers Users Landsat Browse

36. GOFC FIRE - notional, preliminary… • Mid term - 18-36 months • Geostationary data network, e.g. MTSAT • Time dimension • Global views • New systems • GLI, NPP, AATSR, ENVISAT • Multiple brokers (possibly geared towards different user communities/regions) ATSR AVHRR DMSP MODIS TRMM Source Lat/Lon Time Confidence Users Brokers Local GIS Data Producers Brokers Users Landsat Browse Future - multiple paths to data MTSAT - future for SE Asia

37. Forest Cover • Need to develop picture of near/mid term

38. Data access models • Warehouse to user - near term/mid term • Catalog available spatial database servers • E.g. AVHRR - what's available from multiple nodes at a given time, coverage • Publishing and discovery services • Peer to Peer • Cooperative

39. WTF-GOFC Activities • We're working on the Producer/Broker connection first • WTF-GOFC portal • Take the things we have right now - package them for GOFC • Provide documentation, vision, etc. for users, brokers, producers • Provide entry points into demos, systems • Decide which interfaces we're going to use • Data "structures" & packaging • XML, binary files • Metadata elements • Break down the stovepipes

40. Menu discussion • WTF-GOFC menu audience • GOFC users • Data access • GOFC system developers • Tools, prototypes • Data - need access to data • Tools - need access to tools/components • Interface specs • Technical notes • "Virtual testbed" • Development versions can be shown but • Keep a stable version running for public viewing

41. More details on fire • Near term/further out • What do we want to do? • How do we organize ourselves to get it done? • WGISS menu revisit • Guiding principles • Next steps • Cal/Val group - need different services, not a WGISS activity, not in scope for WTF-GOFC

42. Feedback • NASDA • will provide data access support • wants specific science recommendations about NASDA data in terms of science needs • want technical recommendations about data access, distribution, & metadata mechanisms • See NASDA presentation • ESA • Provide support for fire data access • Bringing Ionic WFS/WMS capability in-house • Would prefer rationale for providing GML output - I.e. what's the advantage for users • Provide catalog access to quicklook data, would like to discuss form of request/response • Provide access to ATSR data (quicklook, L1) • MODIS - near-real time service within 12 months (L1b) - wants feedback/requirements

43. Feedback cont. • USGS • Collaborate with CCRS (Landsat 7 - Canada) • Interface to access EDC holdings - L7 via ECS, …, do we create another interface or use an existing one? • Feb 01 - WMS compliant output of Earth Explorer inventories • Looking at pricing for data "series" • DLR • Services - online delivery, dataset tailoring, on-demand processing, "interactive post-processing" • IDL on the Net (ION) - public • Multidimensional DBMS (layered on top of Oracle) - internal • MODIS processing (mid term) • AVHRR, NDVI for Central Europe • CCRS • Provide GOFC portal for data discovery • Provide fire data (currently .e00) • Canada coverage of Landsat 7

44. Rest of week • Tuesday PM • OGC issues • OGC Web Site - http://www.opengis.org/ • Needs to be made more user friendly • Allan Doyle's site - http://www.intl-interfaces.com/wmt2/ • Has some more user friendly info on the WMT but needs to be expanded (feedback welcome!) • At this point we got into more technical discussions • Data discovery & access - John - insert his slides here… • Demo deconstruction - Space Time Toolkit

45. Architecture Principles • Fundamental principle - framework for interoperable distributed services • Emphasize discovery and access (avoid building a generalized service framework before we have services) • Have to deal with heterogeneity in several dimensions • Databases, systems, policies, scale of operations, size of community… • Facilitiate the inclusion of good ideas from outside • Interface based as opposed to data file based • Non-reusable systems are ok if they implement interfaces • Reusable components are ok if they implement interfaces • Open interface standards are better than internal ones • Questions • Message based vs. API based interfaces? • Will we be better off in 1 year based on choices we make today? Agglomeration oftools and servicesheld and operated by CEOSorganisations

46. SERVICE, OPERATION, REQUEST, INTERFACE, PROTOCOL terminology

47. WTF Guiding Principles • The WGISS Test Facility encourages the open sharing of ideas and concepts leading to enhanced capabilities in data management for its projects and the common good. • The WGISS Test Facility believes that such an open environment is essential for the development and advancement of interoperable systems • It is important that partners bringing pre-existing tools and applications, ideas or architectures are given the appropriate recognition and propriety. • Participants need to make clear any parts that are proprietary. • Any components that are developed with the Test Facility are to be shared jointly by those involved and proper credit is to be given.

48. Wednesday • Demo deconstruction • What tools do we have to work with right now? • Next few slides capture current and near term state of systems. • Things that came up during the demo deconstruction: • There are two parts to each of the systems on the next few slides • "Plumbing" - I.e. how things are connected. The schematics show this • Content - I.e. what data exist, what is produced? The schematics don't show this. • One "easy" way to provide OGC Stateless Catalog access to the holdings is by providing access into the systems for a externally maintained catalog. E.g. SGT catalog being populated by TRFIC metadata via the TRFIC SDE interface. • How to deal with treating data as services and as layers (in a catalog)

49. TRFIC schematic Currently provide Willing to provide User Client Would like to access other regional data via WMS or other queries. E.g. Safari-2000, EDC Landsat. Metadata Queries (mid term = Stateless Catalog) TRFIC Native Interfaces HTML Java XML XML XML WMS CGI CGI WWW Server WWW Server Compressed Imagery ArcIMS SQL Derived Land Cover SDE RDBMS NT SQL SQL Landsat Catalog SGI SDE SDE MS-SQL Data Archive Landsat Oracle JPL

50. CCRS Catalog Services Schematic Client Currently provide GOFC Portal User Interface Web Page Will soon provide Area Time IDN Web API Search Engine GOFC Terms IDN Directory XML Distributed Search CEONet Catalog HTML Collection Level Reusable web client components. E.g. Earthscape II HTTP Simple Search Data Internet Access IDN Z39.50 Geo2.2 V0 Data OGC Stateless Catalog OGC WFS OGC Cascading WMS Data EOSDIS Provide service instances at CCRS for use by WTF-GOFC participants?