1 / 14

Efficient Data Management of 3D and 4D Multidimensional Climate Data in NetCDF and GRIB Formats

This document outlines a sophisticated approach for managing climate data stored in GRIB and NetCDF formats, specifically focusing on 3D and 4D multidimensional arrays. It highlights the dimensions of the datasets, including latitude, longitude, time, and altitude. The methodology demonstrates partitioning techniques, efficient indexing strategies, and data compression to enhance accessibility and reduce storage overhead. It also emphasizes fast indexing capabilities for rapid data retrieval and future enhancements for handling large data volumes in climate modeling.

yin
Télécharger la présentation

Efficient Data Management of 3D and 4D Multidimensional Climate Data in NetCDF and GRIB Formats

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GRIB in TDS 4.3

  2. NetCDF 3D Data dimensions: lat = 360; lon = 720; time = 12; variables: float temp(time, lat, lon); temp:coordinates = “time lat lon”; float lat(lat); lat:units = “degrees_north”; float lon(lon); lon:units = “degrees_east”; float time(time); time:units = “months since 01-01-2012”;

  3. 3D data

  4. NetCDF 4D Multidimensional Data dimensions: lat = 360; lon = 720; time = 12; alt = 39; variables: float temp(time, alt, lat, lon); temp:coordinates = “time alt lat lon”; float lat(lat); lat:units = “degrees_north”; float lon(lon); lon:units = “degrees_east”; float alt(alt); alt:units = “m”; float time(time); time:units = “months since 01-01-2012”;

  5. netCDF storage

  6. GRIB storage

  7. GRIB Rectilyzer • Turn unordered collection of 2D slices into 3-6D multidimensional array • Each GRIB record (2D slice) is independent • There is no overall schema to describe what its supposed to be • there is, but not able to be encoded in GRIB

  8. GRIB collection indexing 1000x smaller GRIB file GRIB file GRIB file TDS Index file name.gbx9 Index file name.gbx9 Index file name.gbx9 Create Collection Index collectionName.ncx 1000x smaller CDM metadata …

  9. GRIB time partitioning TDS GRIB file GRIB file GRIB file GRIB file GRIB file GRIB file gbx9 gbx9 gbx9 gbx9 gbx9 gbx9 1983 Partition index Collection.ncx ncx ncx … … 1984 … 1985

  10. NCEP GFS half degree • All data for one run in one file • 3.65 Gbytes/run, 4 runs/day, 22 days • Total 321 Gbytes, 88 files • Partition by day (mostly for testing) • Index files • Gbx9: 2.67 Mbytes each • Ncx: 240 Kbytes each • Daily partition indexes : 260K each • Overall index is about 50K (CDM metadata) • Index overhead = grib file sizes / 1000

  11. CFSR timeseries data at NCDC • Climate Forecast Series Reanalysis • 1979 - 2009 (31 years, 372 months) • analyze one month (198909) • 151 files, approx 15Gb. 15Mb gbx9 indexes. • 101 variables, 721 - 840 time steps • records 144600 - duplicates 21493 (15%) • 1.1M collection index, 60K needs to be read by TDS when opening. • Total 5.6 Tbytes, 56K files

  12. Big Data • cfsr-hpr-ts9 • 9 month (275~ day run)4x / day at every 5 day intervals.  • run since 1982 to present!  • ~22 million files

  13. What have we got ? • Fast indexing allows you to find the subsets that you want in under a second • Time partitioning should scale up as long as your data is time partitioned • No pixie dust: still have to read the data! • GRIB2 stores compressed horizontal slices • decompress entire slice to get one value • Experimenting with storing in netcdf-4 • Chunk to get timeseries data at a single point

  14. Future Plans For World Domination netCDF-4 GRIB

More Related