There were several factors about the verification process that should be mentioned explicitly. For instance, Storm Data was the source of all verification information. For a warning to be classified a "hit", the standard severe thunderstorm or tornado criteria needed to be met. When necessary, the ten-mile/fifteen-minute rule was used to distinguish between singular and multiple events. Since these were scenarios that ran for a set period of time, simple, objective rules were established for grading post-scenario events. Only events occurring during the scenario times were counted as "misses" if no warning was issued. A warning issued for times after the scenario ended would count as a "hit" or a "false alarm" depending upon verification.

Table 1. Basic verification statistics for DLOC workshop scenarios using both the current NWS (county) and proposed (polygon) verification methods. The preliminary 1999 NWS national statistics (from Office of Meteorology, 2000) are also included for comparison

By using the polygon method, the forecaster would only be given credit for warning the areas explicitly selected using AWIPS. Since the forecaster has to create a polygon to issue a warning, it makes sense to use the polygon to verify a warning. Such a method rewards forecasters who are experienced and skilled at forecasting severe events. The polygon verification method may also benefit warning decisions that are made near geographic boundaries. So how will changing the unit area of verification possibly affect verification statistics? If a forecaster issues a warning for a large storm over several counties, the county method will result in several "county" warnings issued while the polygon method will result in only one warning. Since a storm may (and often does) only produce severe weather in some of the listed counties, changing from the county method to the polygon method could result in a significant drop in false alarms issued by NWS forecasters.

There is a potential downside to such a change, however. The average size of a county in the continental United States is over 900 mi² (US Census, 2000). If several events occur in a single county within a fifteen minute time span, then that would be one event for county verification purposes. When the same situation is scored using the polygon verification method, unless those reports are within 10 miles of each other (at best an approximately 300 mi² area), each event would be scored independently. If a warning hits on all the events in this county, then either verification method would produce a "hit." However, since there is no unit area for verification in the polygon method, not warning for the events in this county would count as a "miss" for each event verses only one "miss" using the county method. As a result, using the polygon method could result in a drop in POD. With practice, however, it is probable that this drop in POD could be minimized.

4. RESULTS

4.1 Analysis of WDM I participants

The preliminary verification statistics for the WDM I students are presented in Table 1. The table has statistics for both verification methods, as well as the most recent national verification statistics for the NWS as a reference. Since the WDM I statistics are for only three separate scenarios simulated by several groups, a comparison between these data and the national statistics must be limited. More meaningful conclusions can be drawn as the training data grow over time.

When compared to the overall NWS statistics, the students perform well. Their POD scores are about 10% lower than the NWS average, but their FAR scores and leadtimes are similar (in the case of tornado warnings only, their FAR score is better than the NWS statistics). As much as real warning environments are simulated during WDM I, there are still significant differences between a scenario and a real event. Some of these differences are significant enough (i.e., knowing that you will probably have some severe events in a scenario, not having the pressures of an actual office setting, etc.) to conclude that although their statistics are similar, NWS interns are likely not as skilled at warning operations as NWS forecasters.

A more revealing comparison is made between the WDM I statistics for the two different verification methods. There is a clear drop in both POD and FAR for both categories of warnings. In the case of all warnings, the POD was cut in half, and both CSI and leadtime by about a third, when verification was scored using the polygon method. Many of these observations fit the changes projected between the two methods, even if it is surprising by how