An algorithm developer’s tool

1. An algorithm developer’s tool Valliappa.Lakshmanan@noaa.gov National Severe Storms Laboratory Norman OK, USA http://www.wdssii.org/

2. A developer’s tool • The Warning Decision Support System – Integration Information (WDSS-II) • A collection of meteorological algorithms for severe weather analysis, diagnosis and prediction • Hail, tornadoes, wind, lightning • An integrated set of loosely coupled tools for: • Severe weather diagnosis • Image processing • Statistical validation • Ground-truth verification lakshman@ou.edu

3. WDSS-II Applications • WDSS-II applications (algorithms and tools) are just executables. • launched on the command line. • In deployed systems through scripts. • Can easily change input to filtered form, or accumulate a different product (such as rainfall rate or hail size) • Applications exist for many tasks: • Image processing (smoothing, dilating, eroding, etc.) • Objective analysis (Cressman, Barnes, Gaussian, etc.) • Scoring grids (error statistics) • Statistical skill based on ground truth w2accumulator –i xmllb:/data/realtime/radar/KTLX/code_index.lb \ -I MaxShear_0-3km \ -o /data/realtime/radar/KTLX/ -r -t “30 60 120” lakshman@ou.edu

4. Creating a new algorithm • An algorithm is essentially a data filter • Takes some data as input • Produces new data as output • The algorithm developer should be able to specify the scientific processing in the middle • Without having to worry about data ingest, data formats, notification, etc. • But provide a library of common computations on the typical data used. lakshman@ou.edu

5. w2algcreator • W2algcreator is a WDSS-II tool • To write the format-independent code for ingesting data into your application. • The algorithm developer writes an XML file specifying the inputs and adaptable parameters. • The algorithm itself is auto-generated! • With a “fill in the blank” for the scientific computation • WDSS-II class libraries can be used for common computations. • Easy to add new input and output formats. lakshman@ou.edu

6. Display • The WDSS-II displays are highly configurable to aid trouble-shooting. • Display of intermediate outputs is easy and convenient. lakshman@ou.edu

7. Example intermediate product • Created with no modification of the display. • Just configuration files. • Algorithm developer marks the radar associated with each detection. • For easy debugging. lakshman@ou.edu

8. The end-result • So, what kinds of algorithms have been developed in WDSS-II? lakshman@ou.edu

9. Single-radar/Multi-sensor algorithms • Some single-radar (multi-sensor) algorithms in WDSS-II lakshman@ou.edu

10. Multi-radar/multi-sensor algorithms • A typical multi-radar deployment of WDSS-II lakshman@ou.edu

11. Relevance to Q2 • Which of those are relevant to Q2? • Some existing severe weather algorithms may be relevant • Probability of hail for identifying radar echoes with potential hail contamination. • More likely: • Developing new algorithms • Building algorithms as data filters • Existing lower-level tools. lakshman@ou.edu

12. Accumulation algorithm • Accumulation of shear to form rotation tracks. • Accumulation could be as: • Maximum • Rate • Total • This tool could be used for rainfall depth from rainrate for example. Six Hour Path of Rotational Shear lakshman@ou.edu

13. Motion Estimation • Uses K-Means clustering and Kalman filters 30 min 30 min Actual dBZ Forecast dBZ lakshman@ou.edu

14. Need for new approach • Traditional centroid tracking • Accurate at small scales, but not at large scales • Inaccurate when storms merge or split • Possible to extract trends from the information • Flow-based tracking • Cross-correlation, Lagrangian methods, etc. • Are accurate at large scales, but not at small scales • Not useful in decision support because trends of storm properties can not be extracted lakshman@ou.edu

15. K-Means clustering • K-Means clustering is a hybrid approach • Cluster the input data to find clusters • Like centroid-based tracking methods • But at different scales. • Track the clusters using flow-based methods (minimization of cost-functions) • Like flow-based methods • Does not involve cluster matching (e.g: Titan) lakshman@ou.edu

16. Example clusters • Two different scales shown • Both scales are tracked lakshman@ou.edu

17. Extrapolation • Smooth the motion estimates • spatially using OBAN techniques (Gaussian kernel) • temporally using a Kalman filter (assuming constant velocity) • Repeat at different scales and choose scale appropriate to extrapolation time period. lakshman@ou.edu

18. Trends • The clusters can be used to extract trends of any gridded field. • Configurable to extract minimum, maximum, count, sum, time-delta, etc. of gridded fields within cluster • Even fuzzy combination of multiple fields • Extremely useful for research! • Statistical properties of storms • Changing drop-size distributions with time • Which clusters are convective? • Trends in rain-rates … • Trends in cloud-top temperatures … lakshman@ou.edu

19. Polygon statistics • Using cluster trends is useful for deriving storm properties. • What about extracting statistics around a fixed location? • Maybe around rain gages? • WDSS-II has a tool to provide polygon statistics from any gridded field(s) • The polygons can change with time (e.g: weather service watch areas) lakshman@ou.edu

20. Quality Control Neural Network (QCNN) • Developed for MDA • false alarms in non-storm echo. • With QCNN, shows over 90% reduction in the non-storm MDA false alarms and zero change to detections within storm echo. • The same QC technique would be useful in estimating precipitation as well. • Based on local statistics of reflectivity, velocity and spectrum width fields, vertical statistics and morphological image processing • Handles AP/GC, radar artifacts and some biological signals. • Neural network for optimal combination of inputs. Before QC After QC lakshman@ou.edu

21. Current uses of WDSS-II in the NWS • WDSS-II is a leading edge system • Provides capabilities not yet in the “official” National Weather Service systems. • The Storm Prediction Center • defines daily threat areas • launch a WDSS-II domain • automatically configures the data ingest and starts the algorithms. • NWS forecast offices • WDSS-II products are converted into AWIPS format and piped the AWIPS displays in several NWS forecast offices. • But the AWIPS display is too restrictive. Therefore … • The 4D WDSS-II display is to be implemented as a separate app on AWIPS but controlled from within D2D. • Concept of algorithm development capabilities • Being considered for next redesign of AWIPS lakshman@ou.edu

22. In summary • How can WDSS-II be useful in Q2 • As an algorithm development toolkit • Multi-sensor inputs in real-time and for archived cases • But limited to user workstations • JADE will provide web-based capabilities. • Individual tools • Objective Analysis tools and other low-level tools. • Image processing filters • Quality control of radar data • Motion estimation and extrapolation (short-term QPF) • Storm statistics • Polygon statistics • Please visit this website: • http://www.wdssii.org/ lakshman@ou.edu