PDS Reference Architecture & Standards

1. PDS Reference Architecture & Standards Emily Law NASA ESDSWG, October, 2009 Standards Process Group (SPG)

2. Outline • IEEE 1471 • Reference Architecture • Key Architecture Principles • High Level Architecture View • Mission System End to End • Earth Science • Planetary Science • PDS View • Process Architecture, Standards & Technologies • Data Architecture, Standards & Technologies • System Architecture, Standards & Technologies • Standards Process Needs for Decadal Missions

3. IEEE 1471 • An IEEE Standard for describing the architecture of a software intensive system • Establishes a framework and vocabulary for software architecture concepts • Provides definitions • Provides a Meta-Model for architecture description • Emphasizes an architecture should address a system’s stakeholders’ concerns • Asserts that architecture descriptions are inherently multi-view and modular • Provides guidance for capturing architecture rationales, design and evolution

4. Reference Architecture • Reference Architecture is decomposed into three core pieces: • Process Architecture • Describes polices, system processes • Data Architecture • Describes the information model, data behaviors and relationships • System Architecture • Describes system layers, services, and/or components • Each of the above can provide a set of deliverables to guide system development and operations

5. Key Architecture Principles • Loosely coupled components • Reusable • Abstracted & common service interfaces • Model independent • Technology independent • Distributed

6. Mission System End to End View Science Processing Center 1 Archive & Distribution 1 e.g. PO.DAAC, PDS Mission #1 Science Processing Center 2 Archive & Distribution 2 Distributed Data Analysis (Subsetting, Gridding, Transformation,Modeling) Mission #2 Users e.g. SMAP, DESDynI, Cassini Other Data Sources (e.g. NOAA, ESA) New Area: Infrastructure / System to support Analysis of Distributed Data

7. Research Education Value-Added Providers EOSDIS Data Centers* REASoNs/ MEaSUREs Earth System Models International Partners Decision Support Systems Earth Science View Science Data Processing, Data Mgmt., Interoperable Data Archive & Distribution Distribution and Data Access, Flight Ops Data Capture, Initial Processing, Backup Archive Data Acquisition Data Transport to DAACs Tracking & Data Relay Satellite (TDRS) Spacecraft W W W ECHO* NASA Integrated Services Network (NISN) Mission Services Data Processing & Mission Control Interagency Data Centers ACCESS Ground Stations ACCESS Science Teams (SIPS)* Polar Ground Stations Measurement Teams *EOSDIS Elements

8. Distributed ESDS View GSFC GLAS, MODIS, OMI, OCDPS NCAR, U of Col. HIRDLS, MOPITT, SORCE NSIDC DAAC SEDAC LP DAAC CDDIS ASF DAAC 1 GES DISC JPL MLS, TES 1 1 1 1 1 OBPG 1 1 1 2 1 1 7 1 1 8 1 2 2 1 7 4 1 1 1 2 1 1 1 1 1 LAADS 1 1 1 1 2 1 3 1 9 1 6 1 1 PPS 1 1 1 1 1 ASDC 1 1 San Diego ACRIM LaRC CERES, SAGE III GHRC PO.DAAC GHRC AMSR-E, LIS ORNL DAAC KEY EOSDIS Data Centers Related Data Providers Measurement-based Systems 29 MEaSUREs Science Investigator-led Processing Systems (SIPSs) 17 41 ACCESS REASoN

9. Planetary Science View Relay Satellite Simple Information Object PlanetaryData System Spacecraft / lander Science Data Archive Primitive Information Object Primitive Information Object Science Information Package Science Information Package Science Data Processing Science Products - Information Objects Science Information Package Telemetry Information Package Data Analysis and Modeling Science Information Package Planning Information Object Instrument Planning Information Object Science Team Data Acquisition and Command Mission Operations Instrument /Sensor Operations • Common Meta Models for Describing Space Information Objects • Common Data Dictionary end-to-end

10. Distributed PDS View

11. PDS Reference Architecture Reference Architecture Process Architecture Data Architecture System Architecture Information Model Policy Client Presentation System Process Repository Structure Application Service Data Dictionary System Management Data Processing User Management Data & Resource Management System Service

12. PDS Process Architecture • Policy • Data Delivery and Backup • Data Vetting and Integrity • System Process • Ingestion • Distribution • System Management • Resource Administration • User Management • User Support • User Access

13. PDS Data Architecture Concept PDS Conceptual Information Model prescribe PDS Data Product PDS Label map Mission-Name = XXX Product-ID = YYY … Data Element Definition Data Element Definition Keywords Schema prescribe create describe validate PDS Data Dictionary Data Object

14. PDS Data Architecture • Information Model • Data Types • Data Formats • Metadata • Data Objects • Relationships • Repository Structure • Storage Organization • Directory Tree • Data Dictionary • Global and Local Keywords

15. PDS Data Standards & Technologies

16. PDS System Architecture View Users WebService / API Analysis Visualization Tools Client Presentation Portal Application Service Search Transformation Transport Ground Processing Center Data Processing Data Processing Ingest File Manager Resource Manager Data Distribution Data & Resource Management Data Repository Catalog Metadata Legends: System Layer Data System Component External Entity Interface System Service Workflow Tracking Monitoring Security PDS

17. PDS System Standards & Technologies

18. Standards Needs for Decadal Missions • What architecture views? • What architecture areas? • Which architecture pieces and elements? • What to be considered for recommendation of an optimal set of standards? • Here’s some examples: • Modular Common Reference Architecture Views • Process: Vetting, Cal Val, Citation, User Input, Qualify flag • Documentation: Interface, Testing, Experiment • Data: Metadata, Format, Dictionary, Versioning & Provenance, Error Estimation • System: Security, Web Service, Interface (semantics & protocol) / Messaging, Discovery, Federated Search & Access, Processing Tool Kit, Subsetting, Distribution, Data Assimilation

19. Back Up

20. Data & System Benchmark

21. Distributed ESDS • The common elements of the EOSDIS system include: • Mission Operations (ex: ESMO) • Level Zero Processing (ex: EDOS) • Science Investigator Processing Systems (ex: ISIPS, MODAPS) • Science Teams (ex: ASTER) • Data Centers (ex: DAACs) • Users (ex: broad science community) • The common infrastructure pieces that enable interoperability are: • Networks • Core Resources including physical infrastructure • Standards for science data system software (ex: interfaces; formats, metadata models) • Software systems for reuse • Common processes (ex: testing approach)

22. Specific Recommendation 1 (of 5) • NRC Recommendation 1: Teams of experts should be formed to consider assimilation of data from multiple sensors and all sources, including commercial providers and international partners. • NASA Data Systems can support these efforts by working to make data available and more easy to assimilate through: • Prioritizing sensors and sources (with limited resources, teams of experts decide which data is most important) • Archive, distribution and assimilation of sensor products • Providing references to data that other providers (i.e. interagency and international partners) archive and distribute • Format and documentation standards • Agreements with international partners • Data buys from commercial providers

23. Specific Recommendation 5 (of 5) NRC Recommendation 5: NASA, NOAA, and USGS should increase their support for Earth system modeling, including provision of high-performance computing facilities and support for scientists working in the areas of modeling and data assimilation. NASA Mission Data Systems can support these efforts by providing data in a form that modelers can easily use: formats, documentation, timeliness, error bars, access methods.

24. Data Access and Management (1 of 6) • NRC: Given the breadth of responsibilities of public and private managers and decision-makers, the use of potential applications from the Earth sciences will depend on having easy availability to data that are “accurate, affordable, and accessible”. Many of the decision-makers and other interested parties who need access to Earth science observations and information for addressing important environmental issues are unlikely to be highly-trained Earth science researchers. Non-scientists who use Earth science data and information must have a convenient and intuitive means of access to observations that are relevant to the problem they are addressing. Improving data and information availability and accessibility should include establishing and adopting standardized data and information management practices that foster use and can be understood by the non-scientific user. • Data Systems must make data accurate, affordable and accessible. This includes: • establishing and adopting standardized data and information management practices that foster use; and • providing a convenient and intuitive means for non-scientist users to access relevant observations needed to understand important environmental issues.

25. Carbon Cycle Science Future Needs • Continuous improvement in data access • Especially machine-to-machine interfaces • Ease download limitations, policies to aid in automation • Continuous improvement in data and service discovery • Standardize data formats across multiple missions with community input, but pay attention to backward-compatibility • Select format(s) that are flexible to accommodate future requirements and features without disrupting current operations • Standardize data semantics across multiple missions • Eases data integration and interoperability • Improves automation • Improved simple on-line dynamic data visualization • Quick-look and simple analysis • Multiple datasets comparisons • Standardize data usage policy and make it transparent • Open data policy enhances scientific discovery

26. SPG • SPG Recommendations to date: • Standards: • OPeNDAP, HDF5, HDF5-EOS, NetCDF Classic. GCMD DIF, OGC WMS • Technical Notes: • Aura Guidelines • Backtrack Orbit Spatial Search • SPG reviews now in process or anticipated: • Standards • ICARTT; [netCDF4, CF anticipated] • Technical Notes: • Interoperability with OGC CSW and WCS; Provenance within Data; WCS Server Design; Aura Guidelines Experience, DAP2 to HDF5 Mapping