Grib2

1. Wesley Ebisuzaki (wesley.ebisuzaki@noaa.gov) wgrib2: Wesley Ebisuzaki, Reinoud Bokhorst, Jaakko Hyvätti, Kristian Nilssen, Karl Pfeiffer, Pablo Romero, Manfred Schwarb, Arlindo da Silva, Niklas Sondell, Sergey Varlamov Grib2

2. what is grib2, new features GrADS & grib2 converting from wgrib to wgrib2 inventory grib2 files geolocation read grib2 with wgrib2 write grib2 with wgrib2 Outline

3. What is grib2 • WMO replacement for grib version 1 • Transfer of gridded data between met centers • Software to read/write grib2 • NCO fortran/C libraries, used by many codes • NCAR library, NCL (NCAR command language) • ECMWF grib api, used by many codes • wgrib2 • others: UK? Japan?

4. Why grib1 -> grib2 • Address the limitations of grib1 • Time stamp • Only two bytes to store numbers (255 is ok) • can combine 2 bytes for 16 bit integer (64k is ok) • Files size limitation • Only 128 standard variables, all other are locally defined • Grids only defined to 1/1000 of a degree

5. New features in grib2 • Submessages • Improved metadata: time stamps, level information, variable information • Default grid resolution is 1e-6 degrees but can be specified to be higher • *** Selectable compression schemes ***

6. Gone: kpds(5) .. kpds(7) • Grib1: variable, type of level, level # • Grib2: • variable: table/version, discipline, parameter category, parameter number • New tables to modify variables, ensemble, bias, error, chemical type, statistical processing, etc • level: most used templates require several #s. • wgrib2, GrADS: hide complexity, similar to grib1

7. Forecast timing information • Grib1: 4 bytes: type, time units, 2 bytes • Grib2: size of time stamp is not fixed • No overflows, precise descriptions • Means are handled is systematic manner • in wgrib2 notation, monthly means look like • “124@6 hour ave(6 hour fcst)” • “124@6 hour ave(0-6 hour ave fcst)” • Statistical operations: ave, acc, min, max, rms, etc

8. bash-2.05a$ wgrib2 wafsgfs44.t18z.gribf84.grib2 1:0:d=2011011218:HGT:70 mb:84 hour fcst: 2:5409:d=2011011218:TMP:70 mb:84 hour fcst: 3.1:8037:d=2011011218:UGRD:70 mb:84 hour fcst: 3.2:8037:d=2011011218:VGRD:70 mb:84 hour fcst: 4:16727:d=2011011218:HGT:100 mb:84 hour fcst: 5:22349:d=2011011218:TMP:100 mb:84 hour fcst: Note: 3.1 and 3.2, UGRD and VGRD are in same grib message, share grid description and metadata. GrADS, wgrib2: process by “fields” not by messages. Fortran: extra complexity Submessages are good for irregular grids. Only need to store one copy of the latitude/longitude values. Submessages

9. Bitmaps & Undefined Values • Grib1: uses bitmap for undefined grid value • overhead = 1 bit per grid point • Grib2: use bitmap or special value(s) • All packing methods can use bitmap • Complex packing can use special values • better to use special values if available • GrADS, wgrib2 hides this complexity • fortran code: extra complexity

10. Scan order • Decode a grib field – get a list of grid pt values • “scan order” specifies how you fill the grid • Grib1: we:sn and we:ns are common • Grib2: 16 different orders but 3 are common, we:sn, we:ns and “Glahn” (NDFD) • wgrib2: default is to convert data to we:sn but other options include we:ns and raw but best to keep in we:sn order

11. Grib1, Grib2 – storing grid data • Store scaled integers • Value = (i * 2^E + ref) * 10^D • ref = min value of the field so i >= 0 • E, D are scaling factors for the field • Global model: E = 0, D is specified • Mesoscale: E and D are specified • ECMWF/wgrib2(default): D = 0, E determined by # bits wanted

12. Packing schemes: IEEE • IEEE • store as IEEE real numbers • Allows full precision files (ex. restart files) • No compression • Easy to write (no grib library needed) • Not supported by NCO software

13. Simple packing • Similar to grib1 packing • Store scaled (positive) integers • If biggest scaled integer < 2**N • Store integer as N bits • No compression relative to grib1 • Fast to write and read • common

14. PNG • Compress file by “PNG” compression • same png compression as .png files • Better packings are available • Not recommended

15. JPEG • Uses JPEG2000 to compress the grid data • Best compression for data without undefineds • Slow to write and read • Used by NCO

16. Complex Packing • Start with scaled integer, 3 variants • C1: store values (Vi) • C2: store increments (Vi - Vi-1) • C3: store increments of increments • Split into multiple regions • each region stores packed integer with varying minimum value and size

17. Complex Packing • Heuristic methods to determine regions • Some programs pack more efficiently • Best to use special values rather than bitmap • Undefined encoded as a run of N “special values” • Reading complex packed files is fast. Similar to grib1/simple packing.

18. Complex Packing: file size • C1/C2/C3 is best for noisy/somewhat noisy/smooth fields • Can produce better complex packing by making C1,C2,C3 and choosing the best (C0) • C0 produces smaller files than JPEG for files with undefined values • C0 typically 10%-20% bigger than jpeg

19. Examples: file size • 1/14/2011 gfs.t00z.sfluxgrbf06.grib2 • grib1: 213 MB jpeg: 65.1 MB C0: 65.9 MB * 50.7 MB • 1/14/2001 gfs.t00z.prgb2f06 • jpeg 55.1 MB C0: 56.5 MB * 36.0 MB • 1/14/2011 nam.t00z.edasf06.tm • grib1: 114 MB jpeg: 38.1 MB C0: 34.1 * 24.8 MB • * use best of jpeg, C1, C2, C3

20. Compression: summary • JPEG slow to write and read • Complex packing is fast to read • Complex packing is smaller for files with undefineds otherwise usually 10-20% bigger • Tapes: best compression (jpeg+C0) • Speed: C0 • Multitasking routines: jpeg <-> C0..C3

21. Grib2: common tasks • Display with GrADS • Inventory • Slicing and dicing files • Exporting data • Averaging • Making regional subsets • Making grib2 content: conversion, new data

22. Grib2 & GrADS • Use GrADS v2, g2ctl • Supports ensemble • Supports minutes • Supports 3 time mapping schemes • -0, -b, -verf (use “”)

23. Mapping grib time to GrADS time dimension. :d=2009060500:APCP:surface:324-330 hour acc fcst: d=2009060500 analysis time (-0) d=2009060500+324 hours start of verf period (-b) d=2009060500+330 hours end of verf period (“”) d=2009060500:UGRD:1000 mb:330 hour fcst d=2009060500 analysis time (-0) d=2009060500+330 hours verf time (-b or “”) GrADS: time dimension

24. Reference, analysis time: Use -0 g2ctl -0 ipvhnl.gdas.%y4%m2%d2%h2.grb2 >anl.ctl gribmap -0 -i anl.ctl GrADS examples

25. start of verification time period: Use -b option g2ctl -b gfs.t12z.pgrb2.f%f3 >fcst.ctl gribmap -b -i fcst.ctl must be gribmap v2 end of verification time period: No extra options g2ctl gfs.t12z.pgrb2.f%f3 >fcst.ctl gribmap -i fcst.ctl must be gribmap v2 GrADS examples

26. (forecast) ensemble: files: gep01.t00z.pgrb2cf* gep02.t00z.pgrb2cf* gep03.t00z.pgrb2cf* g2ctl -ens “01,02,03” gep%e.t00z.pgrb2cf%f3 >ens.ctl gribmap -i ens.ctl Can use -b in the above example. GrADS examples

27. Introduction to wgrib2 • wgrib2 was planned • Easy to add modules • People donated modules • Geolocation • Name <-> parameter routines • Update of tables from NCO web pages • mysql export • netcdf export

28. wgrib2 changes • Each option corresponds to a subroutine call • Order is important • command line can look like a small program • Many options, to view • wgrib2 will list main options • wgrib2 -help all will list all options • Functionality • Geolocation, if, for, conversion to other formats, writing grib2, interpolation, multitasking

29. Inventory • Simple inventory: • wgrib2 -s GRB_FILE • wgrib2 GRB_FILE -s is assumed, no need -4yr • Verbose inventory • wgrib2 -V GRB_FILE • -s and -V are inventory options

30. Inventory • Inventory has changed • old names are mostly the same • new levels, old levels are expanded • years are 4 digits long • time stamps can be more complicated • NO KPDS5, KPDS6, KPDS7 or equiv g2lib info

31. Slicing and Dicing • Grib1: wgrib IN -s -4yr | egrep (...) | wgrib IN -i -grib -o OUT • Grib2: wgrib2 IN | egrep (...) | wgrib2 IN -i -grib OUT • -4yr, -s are gone, now 4 digit year codes • Notice: -grib -o OUT becomes -grib OUT • Minor changes to scripts but will need to debug • Watch out for scripts that write binary/ieee/text files • default is we:sn rather than raw. (we:ns in global)

32. Efficiency: -match, -not • wgrib2 IN | egrep A | egrep -v B | wgrib2 -i IN ... • A, B are regular expressions • 4 processes, read file up to 2x • wgrib2 IN -match A -not B ... • multiple -match and -not options • More efficient: 1 read and 1 process

33. Efficiency: multiple write • wgrib2 IN -match ':TMP:' -grib ht.grb • wgrib2 IN -match ':HGT:' -append -grib ht.grb • wgrib2 IN -match ':RH:' -grib rh.grb • OR with one command line • wgrib2 IN -if ':TMP:' -grib ht.grb \ -if ':HGT:' -grib ht.grb \ -if ':RH:' -grib rh.grb • -if is ended by i/o operation or -fi

34. A need to know the values at specified lat-lon. A need to know the lat-lon of the grid points. -lon LON LAT grid value nearest to lat-lon -ijlat I J values and lat-lon of grid point (i,j) -lola ARGS makes lola file with NN interpolation -csv FILE CSV with lat-lon values -spread FILE CSV with lat-lon values Geolocation

35. wgrib2: exporting data • Many output formats • ieee -ieee OUTPUT.bin big endian IEEE • bin -bin OUTPUT.bin machine (IEEE) • text -text OUTPUT.txt • csv -spread OUTPUT.txtcomma sep. values • csv -csv OUTPUT.txt another CSV • mysql -mysql ... to mysql • netcdf -netcdf ... write netcdf file

36. Exporting to fortran • Use -bin OUT.bin(binary) • Default is -header • open(...,form='unformatted',access='sequential') • If you use -no_header option, get direct access file. • open(..,recl=nx*ny*4,form='unformatted',access='direct') • Scan order is we:sn by default, use -order raw for original order

37. Exporting to fortran • Order and number of the records can change! • Be prepared for potential changes • Specify the records that you want (and count) • Sort the inventory before decoding to order • wgrib2 IN | egrep “:(UGRD|VGRD):(2|3)00 mb” | sort -t: -k3,3 -k4,4 -k5,5 -k6,6 | wgrib2 IN -i -ieee sorted.bin

38. call system('grep ":DPT:surface:60 hour fcst:" ds.td.bin.inv | wgrib2 -i ds.td.bin -bin fort.20',status) if (status /= 0) stop 8 open(unit=20,file='fort.20',form='unformatted',access='sequential') read(20) rdata close(20) Only 1 record to read?

39. Use wgrib2: -ave dt GRIB.OUT dt = N(time unit), ex 6hr, 1dy The records have to be in sequence d=2011010100:UGRD:1000 mb:anl: d=2011010106:UGRD:1000 mb:anl: d=2011010112:UGRD:1000 mb:anl: d=2011010118:UGRD:1000 mb:anl: wgrib2 IN | sort -t: -k4,4 -k5,5 -k6,6 -k3,3 | wgrib2 -i IN -set_grib_type c3 -ave 6hr AVE_OUT Averaging

40. Regional Subsets • -small_grib lonE:lonW latS:latN OUT • Cookie cutter of existing grid • -ijsmall_grib i1:i2 j1:j2 OUT • Cookie cutter of existing grid (i,j) coordinates • -new_grid A B C OUT • Uses IPOLATES like copygb • A, B, C are new grid definition

41. -new_grid • subset of copygb • calls NCEP iplib library • not all functionality of iplib is supported • -new_grid is optional, not in opn version

42. Type of interpolation -new_grid_interpolation TYPE bilinear, neighbor, budget etc Interpolation parameters -new_grid_iopts I:J:K iopts(i) array Type of winds (a must) -new_grid_winds TYPE earth, grid Notes: bilinear interpolation is default Must set the type of winds (earth/grid) using -new_grid

43. -new_grid latlon lon0:nlon:dlon lat0:nlat:dlat OUT.grb wgrib2 IN -new_grid_winds earth -new_grid latlon 0:360:1 90:181:-1 OUT Some NCEP grids are built-in: wgrib2 IN -new_grid_winds -new_grid ncep grid 3 OUT input projections: latlon, mercator, polar, lambert, gaussian output projections: latlon, gaussian, lambert, nps, sps -new_grid: example

44. RPN Calculator

45. Why a calculator? • CSV: change units from Kelvin to Celcius • Net flux = downward flux – upward flux • wind speed = sqrt(u*u + v*v) • (3-6 hour prate) = 2*(0-6 hour prate) – (0-3 hour prate) • growing degree days • 500-1000 mb thickness = hgt(500mb) – hgt(1000mb)

46. RPN calculator • old fashioned, stack based calculator • 10 registers 0..9, stack is 10 deep • works on grids • operators +,-,*,/,pop,exc,trig,sqrt,log,exp,etc • ex. convert from K to C: -rpn “273.15:+”

47. The calculation replaces the grid data read from the grib file. The new data can be written out with the standard wgrib2 output operators except for grib data, you have to use -grib_out. Trivial example: changing sign of LHTFL wgrib2 -match “:LHTFL:” -rpn “-” -grib_out neg_lhtfl.grb Saving calculator results

48. wgrib2 IN.grb -match GRD \ -if “:UGRD:500 mb:” -rpn sto_1 -fi \ -if “:VGRD:500 mb:” -rpn sto_2 -fi \ -if_reg 1:2 -rpn “rcl_1:sq:rcl_2:sq:+:sqrt:clr_1:” \ -set_var WIND -grib_type jpeg \ -grib_out out.grb calculating wind speed

49. Changing metadata • previous slide: -set_var WIND • many options to alter the metadata • -set_var HGT • -set_date 2011010212 • -set_ftime “12 hour fcst” • -set_lev “200 mb” • -set_metadata FILE

50. Writing out new grib2 file • Select compression • -set_grib_type (jpeg|s|c1|c2|c3) • Compress data and write new grib message • -grib_out NEW.grb • Examples of combining -rpn, -set_* and -grib_out to write a new grib message are on web site