Wednesday, March 9, 2011

Forecaster Grades for the March 8-9 Snow Event (Non-Event?)


Our standard disclaimer: The grades we provide are not scientific. However, they represent our best effort to assess the performance of Twin City weather outlets. Our grades are based on established grading criteria and we maintain forecast histories for each event. It might also be helpful to read our “Weather Watcher’s Bill of Rights” to get a sense of our perspective.

In recent memory, it’s hard to recall a snowstorm that was advertised so early and with such ferocity. It’s also hard to recall a storm that felt so short in comparison to initial expectations.

Last Tuesday evening (March 1), most weather outlets identified the possibility of a big storm. From Thursday to the first part of Friday, “storm hype” was in full force (see below). Shortly after that, forecasters began to sense that the storm was tracking to the southeast and not likely to make a direct hit on MSP. However, even after adjustments were made to the initial “whopper” forecast, the storm was still an underachiever in comparison to final forecasts. And while TMF does not specifically evaluate forecaster performance in locations outside the Twin Cities metro, it seems clear that “the big hit” never really hit anyone.

Prior to the grading section, we wanted to take a more detailed look back at the so-called hype. Several forecasters, particularly in the last few days, have seemed to distance themselves from the hype produced by “some other forecasters.” We reviewed the progressive forecasts and, depending on how you define hype and what you perceive to be a prediction vs. a “mention of possibility,” it seems that the “other forecasters” were more or less everyone. Here’s “the height of the hype” from each weather outlet.

WCCO: One model said 12-16
KSTP: Could spread 6-12 inches our way
Fox: 10-15 inches
KARE: Looking very much like the other big storms this winter
Strib: 13+ inches
MPR: This is a perfect track to dump heavy snow in MN, including the metro

In the end, the snow event of March 8-9 totalled exactly one inch of snow at the MSP airport.

Here is our forecaster review and grades:

MPR
MPR was very deliberate in the timing of its first issuance of snowfall estimates. In fact, MPR included 225 words on its Monday morning blog post devoted to discussion of their snowfall prediction policy (24-48 hours prior to an event). It spoke to readers in a clear, down-to-earth manner.

We were disappointed with the rather long time (three days) without at least a short update on the MPR blog during a time when prospects for a major storm were changing and new information was available.

Early on Sunday morning, MPR said there was an 80% chance of a plowable event that included the Twin Cities. Obviously, this did not transpire. That said, MPR was the first to mention a lower range of snow totals and also emphasized the low end of 1-4-inch prediction before the storm started. Our grade for MPR is a B-.

KARE
Conservatism was king for this storm. The less early hype given to the storm, the better the grade, in our minds. KARE’s policy of not providing snowfall accumulation estimates until 24 hours prior to a storm served them well this time around.

KARE made an exception to its extended outlook policy (which generally seems to be five days, except in the case of weekends) by mentioning the storm a week out. They commented on the similarity of the look of the as-of-yet-unformed storm to the big storms of this winter, thereby suggesting a big one might be on the way.

KARE’s first snow totals after the general “few inches” were for 3 to 5 inches. Their final 2-4 inch snowfall prediction for the metro (which we extrapolate to mean 3 inches for airport) was high and still a significant variance for this sized storm. We give KARE a grade of B-.

KSTP
KSTP was an early rider on the big-snow wagon with the prospect of a foot mentioned in a blog entry on Wednesday evening. However, their cause was helped out to a degree by their use of confidence ratings (something regular TMF readers know that we’re big on). On Thursday, they gave a 10 (on a 10-scale for confidence) to the possibility of “measureable snow” with this snow event (which happened, though a mini-ruler was sufficient). Their 7 on the same scale for a “plowable snow” reflected a strong conviction, yet not a certainty. However, just as surely as a .300 hitter in baseball gets a hit, their prediction was wrong. (Think about that one….)

Still, on Saturday morning, KSTP reported that “someone in the Midwest will get nailed.” We interpret “nailed” to be upwards of 6-12 inches in this context, that did not happen. KSTP’s final prediction of an average metrowide accumulation 3 inches was considerably high.

While KSTP’s use of social media and confidence ratings continues to set the standard, its forecast this time around was lacking in accuracy. We gave them a D+.

WCCO
WCCO’s effort on this storm brought attention to the use of “tossing out” model numbers. It’s our feeling that if a forecaster, particularly a television forecaster, voices possible snowfall amounts in any way – as an actual prediction or as a possible output from a model run – the attention span-challenged viewer is going to walk away remember one thing: the largest snowfall amount they heard. On Thursday evening, WCCO cited that one model spit out 12-16” and another 15.5”. Related to that, we got a kick out of the Saturday 6 p.m. weathercast when Mike Fairbourne refused to answer the anchor’s request to provide weather model information and snowfall projections. Fairbourne responded, in his typically gentle way, “I’m not tipping my hand. It’s still three days away!”

Like most other weather outlets, WCCO overrated the storm even after the storm’s likely movement to the southeast was widely accepted. They suggested the snowfall would be a plowable event as recently as Saturday night. Our grade for WCCO is D.

FOX
FOX was a little slow out of the gate on this one. Last Tuesday evening, it was the only station to not mention the possibility of snow in its extended forecast for Sunday through Tuesday (recall that in addition to the just-ended snow, several inches of snow also fell on Sunday). Yet they ended up riding the same roller coaster as everyone else, suggesting 10-15 inches of snow at one point on Friday. FOX’s grade is a D.

Star Tribune
The old saying, “You live by the sword, you die by the sword,” seems appropriate in this case. When you throw out 13”+ in writing and it actually happens, you’re a rock star. When it doesn’t, you become the object of considerable criticism.

Partly given the forecaster’s early and large “prediction” and partly because the Star Tribune chose to feature the weather blog’s brassy and flatline “13+ inches of snow next week” headline on its highly visible home page, an expectation level was set for an entire metropolitan area, if not a region. And while the Strib presentation was sprinkled with disclaimers such as “this is a forecast; forecasts are often wrong” it somehow got lost in the context of predictions that some Strib readers called outrageous.

Clearly, this was not a winning forecast, nor did it seem to be handled in a particularly responsible fashion from a journalistic point of view. Our grade for the Strib is an D-.

NWS
We track the NWS by simply noting the text forecast rather than getting into the official NWS discussion. However, based on the straight day-to-day forecasts, the NWS was steady. However, it also suffered from snow overprediction, forecasting 3-4 inches as of Monday afternoon. The NWS grade is a B.

Agree? Disagree? Share your thoughts.

10 comments:

  1. What little snow we got in the driveway melted by the end of the day. Pretty uneventful.

    ReplyDelete
  2. Far be it from me to "white-knight" Paul Douglas, but: Who put that "13+" up on the front page, out of context and with no follow-up? I'm betting it wasn't PD.

    ReplyDelete
  3. Very good point, Dennis. Perhaps a slight grade amendment is in order.

    ReplyDelete
  4. I had the same thought, must have been the web page editor

    ReplyDelete
  5. I wonder if TMF should grade the actual forecasters, instead of the station. I have been noticing a great deal of variation among the forecasters within a single outlet, sometimes wider than forecasters across different stations.
    This reflects in my opinion a sharp difference in actual competence. People like Steve "somebody in the Midwest is gonna get nailed" Frazier of KSTP is an embarrassing experience to watch for his sheer incompetence.
    I suggest that TMF would disregard whatever he says when assessing the grades. The guy really does not know what he's talking about.
    But to be fair: having enough weather training to be able to interpret myself model outputs, I have to say the models were really all over the place with this one, even 24 hours out. The 00z run of the NAM on Monday night, was still printing out 5-6" of snow for MSP. When the NAM is busted 24 hours out, I can see the frustration of the forecasters. THis was really a tough one to get right.

    ReplyDelete
  6. The challenge we'd have with grading the forecasters, which I think definitely has some merit, is that how do you gather their forecasts when they're not working? Suggestions?

    By the way, I know that these are tough grades but I guess that should be expected with such a tough forecast? The other question, I suppose, is whether forecasters should be graded simply against each other vs. in absolute terms. Perhaps another major meteorological dilemma of our times. :-)

    Appreciate the input.

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. Bill - very well written grade analysis.

    No need to defend PD - he puts himself out on a limb all by himself.

    I'm willing to accept the models are going to be wrong at times, so I can give the forecasters some slack over the final 48 hours prior to the onset of the Tuesday night snow.

    But the across the board 7-8 day hype is what I found disapponting. This is a what-have-you-done-for-me-lately-busniess, and I'm sure this is not how the local mets wanted to end the season. And no, it is not okay to say after a missed forecast, "At least we got lucky" (meaning we didn't want or need more snow). It is okay to say, "We (gosh forbid someone says "I") missed this one". And I'll fall out of my chair if an on-air met ever says, "sorry".

    Anyway, funny how the Sun night into Monday (March 6-7) storm ended up being the bigger storm @ MSP. To me the gold standard is still the NWS. Yes they can be wrong, but they have almost no reason to hype. With that, I was dissapointed by their 7 day lead "weather graphic" as well as hazardous weather outlook depicting "most likely" heavy snow band bullseye directly over the Twin Cities. I think their "B" grade was generous.

    I've been monitoring this site for about a month now, and do enjoy the thoughts and the insight by the likes of randyinchamplin, bigdaddy, novakweather, etc. My advice/suggestion to the regulars - create an ID on blogger. Your actual name is not tied to it (unless you want it to be). Thanks.

    ReplyDelete
  9. As an addition to what I just posted, some kudos to PD on his most recent post on his site. He actually did come out with an apology. If anyone doesn't know where his site is, do a search for "Paul Douglas Weather Column" which is also on blogspot. PD also has an analysis as to what happened and states he is thinking of presenting some of his info with a better disclaimer. Paul, if you read this, very well thought out.
    May we all get a good night sleep now with nothing major on the horizon...

    ReplyDelete
  10. MN Weather Fan: Thank you for the comment above, I don't know if you saw my insanely cold forecast that I posted on the previous thread, (at least it is compared to media mets) but after looking at the 0z run of the Euro on the E-wall site, I'm sticking with it...I think our locals are using the gfs/nam when the NWS office in Duluth is using a blend of the euro/gem...oh well

    ReplyDelete