WDSS-II Training Module IV

1. WDSS-II Training Module IV Algorithms and Tools

2. General Notes • Output from WDSS-II applications may be shared across multiple machines • Any application can use the output of another application as input • The wg display is an example of this • It provides input/launch to the “Filter” algorithms • It uses products from other algorithms • Real-time and “data playback” modes are essentially the same modes of operation

3. WDSS-II application types • Data ingest applications (“ingestors”) • Single-source algorithms • Usually single-radar applications • Multi-source algorithms • Combine input data from multiple sources of one or more instrument types • General use tools • Data filters, objective analysis tools, data remapping, data converters, verification tools, etc.

4. WDSS-II primary data types • LatLonGrid: geographic projection • Equal spacing in degrees latitude and longitude • RadialSet: cylindrical projection • Accommodates any number of radials with variable radial widths • PolarGrid: an indexed RadialSet • DataTable: for point data • Trends • tracks • CartesianGrid: equidistant projection • equal spacing in N/S/E/W directions Other types to be described in a later presentation

5. Data ingest • Data-ingesting programs read “raw” data files and convert them to one of the internal WDSS-II formats • New input types are easy to add • Maintains a consistent internal structure for data sharing among applications

6. ldm2netcdf Dashed lines represent optional inputs, data sources, or applications Reflectivity Velocity Sp. W. w2qcnn Other optional algorithms Reflectivity OR ReflectivityQC ReflectivityQC netssap w2hail w2vil CellTable MesoTable TvsTable MESH POSH MESHTracking Echo Tops (H_*) VIL Comp. Ref. WDSS-II Real-time data flow Single-radar products Legend satellite data* RUC analysis data (grib) WSR-88D data (level 2) Data sources are in ovals *Satellite data are required to be in netcdf format. Applications are in boxes gribToNetcdf nse1 w2cloudcover swatScit2D w2circ Scit2D (table) AzShear Divergence AzShear layers 1If nse is not used as an input, then PolarHail.xml and ssaparm.dat should be updated twice daily. It is highly recommended to use nse data if accurate hail guidance are desired.

7. The most-used single-source algorithms • w2qcnn: quality control neural network • May use radar-only data, or radar plus cloud cover information • Output: ReflectivityQC & ReflectivityQComposite • http://cimms.ou.edu/~lakshman/Papers/qcnnjam.pdf • w2circ: radial velocity derivatives • Produces rotational (AzShear) and divergent (Divergence) shear fields for every tilt • Also produces layer maxima (e.g. 0-3 km MSL)

8. The most-used single-source algorithms • nse: near-storm environment • Parameters are derived from the RUC model analysis • Provides input to other algorithms • Output similar to SPC mesoanalysis web page

9. Other single-source algorithms • w2hail: hail grids and echo tops • w2vil: VIL and composite reflectivity • netssap: the original SSAP • MDA, TDA, SCIT, HDA, DDPDA • Requires copy of *.dat configuration files in working directory • dealias: independent executable of WSR-88D build 10 dealiasing • Note that dealiasing is usually done automatically in data ingest process for WSR-88D data (ldm2netcdf)

10. Dashed lines represent optional inputs, data sources, or applications WDSS-II Real-time data flow Multi-radar products Legend nse* Data sources are in ovals Scit2D (x N radars; or from w2merger) AzShear[layer] Reflectivity[QC] (x N radars) *If nse is not used as an input, then MRScitHail.xml should be updated twice daily. It is highly recommended to use nse data if accurate hail guidance are desired. qcinfo Applications are in boxes w2merger w2merger QCTimeInfo MergedAzShear[layer] scit3D MergedReflectivity[QC] MergedReflectivity[QC]Composite VIL products Reflectivity_X1C EchoTop_Y2 HY2_Above_HX1 (“Height Above Isosurface”) MESH /POSH / SHI (Hail) Scit2D (from 3D grids) 1isosurface(C); 2reflectivity value (dBZ) w2segmotion MR_Celltable ClusterTable MergedReflectivity[QC]CompositeForecast (15,30,45,60 min) Windfield w2accumulator (RotationTracks) MESH Tracks (2 hr, 6 hr, etc)

11. w2merger • Multi-radar data merging • 2D or 3D • Continuously updating • The grid is updated each time data are received from any source • Writes output at user-specified time intervals • Any resolution (Vertical/horizontal) • Also runs algorithms on the 3D data field • http://cimms.ou.edu/~lakshman/Papers/w2merger.pdf

12. w2merger preparations: cache • Pre-compute the radars that will sample the grid point (the “cache”) • Makes all computations faster • Beam blockage is considered • Use program “createCache” (once for each radar) • w2merger will create a cache on-the-fly if one is not available, but: • It will not include terrain data • Data will not be processed until the cache creation is complete (which might take a while)

13. w2merger preparations: cache • By default, the cache is stored in ~/.w2mergercache • It might be big! If you are finished processing a domain, you should delete it • A cache may be extracted from a cache with larger spatial extents (“createCache –e”) • Within NSSL: extract from /mnt/radararchive • Another option: createSubdomains – create caches for all radars in the domain

14. w2merger preparations: cache • You may reduce the number of radars that affect a point by running “postprocessCache” • e.g. if you only want the 3 “best” radars to impact the calculation at a point

15. Merging strategies • Different products may require different ways of combination • Set through the ‘-C’ option • Some examples: • Reflectivity: ExponentialTimeAndDistance or Distance • AzShear: MagnitudeMaximum • Velocity: InverseVAD or MultiDoppler • Choose the most appropriate method for the product you are merging. • There are others: see w2merger usage for list • If you need a different merging option, add it!

16. Running merging and algorithms separately • Algorithms may be run each time w2merger writes out 3D grids of reflectivity data • If the merger is CPU-intensive or I/O-intensive, then run the algorithms separately, perhaps on another machine • w2merger option “-C 10”

17. w2merger algorithms(-a option) • Composite or VerticalMaximum • vertical maximum at each lat/lon • VerticalMinimum • vertical minimum product at each lat/lon • AbsMax or AbsoluteMaximum • abs-max product at each lat/lon. The result retains the sign of the maximum. • VIL • vertical integrated liquid product at each lat/lon (assumes that the 3D grid is a grid of Reflectivity) • Includes different integration strategies (e.g. along storm tilt, VIL Density, etc)

18. w2merger algorithms(-a option) • HDA • produces SHI, POSH, and MESH at each lat/lon (assumes that the 3D grid is a grid of Reflectivity). • SCIT • creates 2D storm cell features from the multi-radar grid (assumes a grid of Reflectivity). • LayerAverage or Isotherms • produces Reflectivity at various isotherms (0,-10 and -20C), ReflectivityBelowZero, LowestReflectivity, etc.

19. w2merger supplemental output • MergerInputRadarsTable • Provides information about the current data streams • Age • Tile • VCP • Useful for determining which radars went into the output

20. w2segmotion: storm segmentation and motion estimation • Multiple scales • Can generate statistics based on storm areas • Motion estimates feed back into w2merger for time/space correction • http://cimms.ou.edu/~lakshman/Papers/kmeans_motion.pdf

21. Mr. SCIT (Multi-radar storm cell identification and tracking • “scit3D” executable • Use “-g” option for Scit2D features generated by w2merger • Use “-t” option to ingest grid fields of various parameters that should be added to the output table • Environmental data from RUC analysis • Precipitation rate field • Etc. • Produces “MR_CellTable” output

22. w2accumulator • Take the: • Maximum • Minimum, or • Sum of all tables or grids produced over a specified time interval. E.g.: • 2-hour max MESH = a hail swath • 6-hour precipitation rate integration • 4-hour max of 0-3 km Azimuthal Shear (“Rotation Tracks”) • DataTable, RadialSet, or LatLonGrid

23. Other useful algorithms • w2cloudcover: estimate cloud cover over a region using IR satellite and surface temperature • w2vortdiv: compute vorticity and divergence from a 2D wind field • w2alarm: collect statistics within an earth-relative polygon for any grid

24. Data Converters • w2awipsnc: convert WDSSII netcdf grid files to AWIPS format • w2cropconv: convert and remap any WDSSII RadialSet or LatLonGrid to a LatLonGrid • w2csv2table: convert a CSV file (spreadsheet) to a WDSS-II DataTable • w2table2csv: vice versa

25. Data Converters • w2geotiff: convert a WDSSII netcdf file to a geoTIFF file • A TIFF image file with geographic information tags (for GIS) • w2grib2conv: convert a WDSS-II file to GRIB2 • netcdf2ldm: convert a set of WDSSII netcdf files to WSR-88D level II format • Can replace AliasedVelocity with Velocity, Reflectivity with ReflectivityQC for example

26. Objective analysis / filters • w2smooth: smooth the data using one of many strategies: • Gauss • Cressman • Percent (e.g. median) • Oriented • Ellipse • Various wavelets

27. Objective analysis / filters • w2threshold: Thresholds one field based on another • Example, remove VIL in areas where the IR temperature is > 250K • Various options to smooth (using w2smooth internally) and/or segment field

28. Objective analysis / filters • w2oban: convert point data to a LatLonGrid • w2morph: morphological filters • Dilate • Erode • w2contour: create contours of a data field

29. File manipulation • w2get: copy a file via rssd • w2mirror: mirror all the files listed in an lb to a different machine • Limits the number of users “hitting” a real-time machine • w2simulator: simulate real-time data playback • w2stitcher: stitch together two different domains into one larger one

30. Suggested exercise on archive data • Download KTLX and KINX data from May 20, 2001 from 21:00 to 22:00 UTC from NCDC • Convert it into WDSS-II netcdf format • Run w2vil to produce VIL estimates in rapid-update mode • Merge the VIL estimates using w2merger • What is a valid combination strategy here? • createCache before merging! • Compute VIL from merging reflectivity data • Compare the two VIL estimates • Find their difference field using w2scoregrid

31. Suggested exercise on real-time data • Connect to two adjacent radars that are currently experiencing weather • Look at the 2DConUS index • Overlay the radarsites shapefile • Find LB names from the tensor list • Create cache for domain using createSubdomains. • Extract from /mnt/radararchive • Run w2vil, w2merger and w2scoregrid as described before. • Set up a w2alg.conf to do this.

32. End of WDSS-II Training Module IV • What to do next : • Practice running some algorithms and tools. • You will not be able to follow module 6 (writing a WDSS-II algorithm) unless you are familiar with how WDSS-II algorithms in general work. • Run both a single-radar algorithm and a multi-sensor algorithm. • Run both on archived cases and real-time cases. • Next module: Configuring WDSS-II