1 / 8

Effective Aggregation and Subsetting of Real-time Data in Unidata IDD/LDM Systems

This use case explores the challenges and solutions for serving and managing real-time data streams from Unidata's IDD/LDM systems. Key focus areas include handling diverse datasets such as model, satellite, radar, and station profiles, and the complexities of aggregation and subsetting using netCDF and OPeNDAP data models. The discussion emphasizes the importance of understanding coordinate systems for effective data access and the necessity for clients to perform significant preparation work. We also cover different aggregation techniques and the trade-offs involved in dealing with array index space limitations.

sydnee
Télécharger la présentation

Effective Aggregation and Subsetting of Real-time Data in Unidata IDD/LDM Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Aggregation/Subsetting Use Case: Unidata IDD/LDM Data Ethan Davis UCAR Unidata

  2. Unidata Use Case • Serving data from IDD/LDM data streams • Real-time data: model, satellite, radar, station, profiles, etc. • A lot of data, e.g., several radar data records per second • delete after 7, 30, 45 days depending on data and server

  3. Starting from netCDF data model (array index space) • netCDF and OPeNDAP data models don't understand coordinate systems • Arrays and index space • Sequences with constraints • Lots of limitations when dealing with array index space • Types of aggregation • Join on an Existing dimension • Join on a New Dimension • Union

  4. Problems with Array Index based Aggregation • Data access/subsetting: • Client WANTS to deal with coordinate systems • Client must do some heavy lifting • rolling archive means the mapping between index space and coordinate space is potentially time dependent • Aggregation: • Brittle: Data must be VERY homogeneous (any variation breaks things … and there's always variation in real-time data)

  5. Coordinate System and Data Type Aggregation/Subsetting • Aggregation • Higher-level understanding of datasets allows for improved aggregation. • Not as brittle. • Better understanding of needed metadata changes • Subsetting • Higher-level understanding of datasets allows for services that don't require as much work by client • Grid: OGC WCS and WMS • Point, station, profile: • TDS NCSS, etc. • OGC SOS and WFS (* Outside implementations) • Advantages: • Easier for users/clients • Can better handle real-time/changing datasets

  6. GRIB storage

  7. netCDF storage

  8. GRIB Rectilyzationologicment • Turn unordered collection of 2D slices into 3-6D multidimensional array • Each GRIB record (2D slice) is independent • There is no overall schema to describe what its supposed to be • there is, but not able to be encoded in GRIB

More Related