1. MUM Architecture and Data Flow Global Archive MM5 Data Extractor Satellite Numerical Model: MM5 Stoplight Servlet Stoplight Servlet Global Verification Stoplights MM5 Uncertainty Stoplights Stoplight Table Tag Stoplight Table Tag SREF Stoplight MM5 Data Extractor Probabilistic Ensemble forecasts External Link to At.Sci. RMSE Verification Plots Zero-Hour Analysis / Satellite Overlays Meteograms Mesoscale Error Plots PlotImageGen Servlet MeteogramPlot Servlet Uncertainty Portal Verification Portal Link to Uncertainty Portal Link to Verification Portal External Link to Atmospheric Sciences MVIRServlet Mesoscale Image Archive Mental model of the atmosphere • Goals and steps to reach e.g.Make comparisons Models to satellite Models to models Avoid • Calculate differences • Maintain results of calculations & comparisons • Make decisions Sources of information Evaluation techniques Forecast Rules of thumb Set routines Knowledge about model biases & strengths Long Term Memory Working Memory The MURI* Uncertainty Monitor (MUM) David W. Jones, Applied Physics Laboratory, University of Washington, Seattle, Washington and Susan Joslyn, Department of Psychology, University of Washington, Seattle, Washington Introduction Prototype Solution: MUM (MURI* Uncertainty Monitor) A software prototype that assembles, processes, and visualizes uncertainty information for weather forecasters. Problem: Navy weather forecasters rely on numerical models to produce forecasts, but also have to consider the level of uncertainty in the models, which can be crucial in the tactical decisions the military has to make. • Frees forecasters from the computational demands on their working memory. • Lets forecasters use models and techniques appropriate to specific location and situation. • Alerts forecasters to problems and encourages thorough evaluation. • Allows quick assessment of model uncertainty. Approach: University of Washington researchers studied the forecasting task and the users who perform it, then built a tool to increase the forecasters’ ability to evaluate uncertainty. UW researchers asked the question: Would Navy forecasters use additional uncertainty information? To answer the question UW researchers conducted two studies: MUM Interface – • Information presented in past-present-future on left control panel, bottom center and right control panels. • Users select information needed on the control panels and a visualized representation appears in the center. • Task analysis revealed forecasters spent most of their time reviewing model initialization. Thus, MUM’s default is the current model initialization field overlaid on top of the most current satellite picture. Study 1 - Think aloud Verbal Protocol Analysis of Terminal Aerodrome Forecast (TAF) Forecasters thought aloud through the generation of their forecast to explain what they were doing and how they did it. They functioned under severe time pressure and often had to do other tasks simultaneously while producing their TAF. Study 2 - Post-TAF Questionnaire Forecasters filled out questionnaires after their TAF indicating their sources of information and rating the numerical models’ performance. Researchers wanted to find out if they increased their evaluation techniques or used additional information sources when the models were judged to be less reliable. MUM Architecture – • Based on Java Server Page technology that only requires a browser to read and interact with the system. • Model data is produced at the lowest tier, post-processing at the second, and user interaction at the top tier. Study 1 Results: Forecasters used few sources of information, mostly models, and already evaluated model uncertainty on every forecast, but relied on rules of thumb to avoid the computationally intensive procedure of model comparisons. Study 2 results: Forecasters’ routine varied little from one forecast to the next. There was no correlation between model ratings and forecasters’ evaluation strategies or their use of other information sources such as satellites or buoys. The ensemble spread meteogramdisplays information about the MM5 ensemble performance for a single geographic location and parameter over a four day period. The most recent 00 hour prediction lies at the center, marked by a bright vertical line. Why? Human information processing tells us that people have large long-term memory capacity, but limited working memory capacity, which is aggravated under time pressure and the task switching demands common in the Navy. • Major Findings: • Naval forecasters are concerned about model uncertainty, but tend to avoid computationally intensive procedures such as • Examination and comparison of multiple models • Comparisons to multiple sources • Head-to-head comparisons between models • Adjustments to the forecasting process to the perceived model uncertainty • Evaluation of model performance over previous few days MUM with ensemble spread meteogram Conclusion MUM will continue to be used to test methods of presentation and user interactivity toward the goal of improving forecast quality, timeliness, and usefulness. Forecasters can now use probabilistic information in new and innovative ways. Forecasting tasks that are working memory intensive are either offloaded to long term memory or avoided For more information, contact: David Jones, APL-UW (206-543-3236 dwjones@apl.washington.edu) * This research is supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745. The support of the sponsor is gratefully appreciated. The MUM system is based on Java Server Page (JSP) and servlet technology. It is hosted at the Applied Physics Laboratory (APL-UW) on a Linux system running a Tomcat server. The model data used in the system comes from the UW SREF. This includes the global fields used for the SREF boundary conditions and the individual ensemble members of the SREF. This data is stored and archived on the APL-UW server.