Tuesday, February 22, 2011

Forecaster Grades for February 2011 Monster Storm


We know that the grade is the pop culture hook, but we think as much or more attention should be paid to the overall analysis and the forecast process. The grades we provide are not scientific. However, they represent our best effort to assess the performance of Twin City weather outlets. Our grades are based on established grading criteria (read ‘em!) and we maintain forecast histories for each event.

As the storm evolved in the past week, we consulted daily with Tom Novak, a Minnesota-based meteorologist, who provided an objective, professional assessment of information generated by various weather models, the same ones available to all meteorologists. We are deeply indebted to the significant chunks of time that Tom spent with us. His excellent work may be followed on Twitter at @NovakWeather.

Generally speaking, the snowstorm of Feb. 20-21 performed more or less as expected by the local forecasters based on final pre-storm predictions. It came in a little later than expected, struggled initially to penetrate the northern suburbs, and experienced an extended lull before today’s unexpectedly lengthy coups de grace, but in the end it was more or less what was promised, particularly in the central metro (i.e., MSP airport), where we focus our grading. The final two-day snowfall total at MSP was 13.4 inches (as of this writing, anyway; we think it’s done), which fit pretty squarely into all forecaster ranges. So to the casual observer and in the big picture, the forecast of an epic snowstorm rang true.

However, upon closer examination of the details of each forecaster’s prediction and their assessment of the storm in the days leading up to it, significant differences are noted. These differences are reflected in the grades and discussion below. (The complete history of each forecaster’s prediction for the storm can be found here.)

WCCO
The specter of the weekend snowstorm crossed WCCO’s radar as early as Monday night. Their weather graphic depicted “snow late” for Sunday, but there was no audio mention of it. It was essentially the same on Tuesday when the graphic depiction read “snow south,” but again there was minimal, if any, elaboration on the possibility of a significant storm. By Wednesday, WCCO gave the storm a bit more respect, mentioning that “one model says 7” but most keep it far enough south.” By Thursday, WCCO saw a possible mix and by Friday it was laying out all the model outputs, which “suggested snow amounts of 2, 4, 9, 11 and 15 inches.” By Friday afternoon and evening, WCCO was on board the big bus.

All things considered, WCCO’s performance was average. We were disappointed that they didn’t use social media to provide timely updates and felt an additional webcast or two were in order during the day on Saturday. And while WCCO certainly can’t be accused of anything less than full disclosure with its mention of the various model-produced snow amounts throughout the forecast period, we think the impression that’s left with viewers is one of confusion when so many amounts are thrown out without real perspective or insight. We’d have preferred to see more of a professional stand taken. C+

KSTP
This was a true championship performance for KSTP. They nailed this snowstorm from start (more than a week out) to finish, generally maintaining their forecast conviction when slight changes in weather models caused others to literally go south with their predictions. In fact, this was a critical point in the process: at midweek, our consulting meteorologist did not see the justification for bringing the storm south as many predicted. KSTP earned the prize money by maintaining its belief in the ultimate path and character of the storm.

And on the communication front – if you haven’t noticed by now, we’re big on using all the information tools available today to keep the audience informed even when they’re not in front of a TV at 6 and 10 – KSTP really shined, fully exploiting the ability of Twitter, Facebook and webcasts to provide information in a far more timely and interactive way than virtually all other local weather outlets. We also appreciated KSTP’s imaginative use of graphics to reflect confidence levels in various snowfall outcomes.

On Thursday evening, a time when the television weather outlets were calling for mixed precipitation or otherwise minimizing accumulations, KSTP was steadfast, calling for the heaviest amounts to fall in a band through the Twin Cities metro. Had you planned your weekend on Thursday night and only watched the television weather outlets, KSTP gave you the best idea of what was to eventually unfold. A+

FOX
The friendly folks at Fox weather have seen better days. On Monday night, there was no mention of the possibility of snow. They got on board by Tuesday, noting “a decent system for Monday.” But by Thursday evening’s 5 p.m. news they were off the bus, telling us there was “not a great chance of a big snow.” As the storm began to grow in potential over the weekend – and if you didn’t happen to catch a FOX television news segment – the Fox website was not much help, showing outdated weather forecasts at a time when the situation was rapidly changing. Not good. Fox cleaned up nicely toward the end with a solid forecast that was a touch more specific than others’ and with ranges (referring to the span of the ranges) that we thought were reasonable. D+

KARE
We like the KARE forecasters, we really do. But gosh it’s hard to pin them down. Their philosophy, which was on full display for this storm, is very noncommittal, so much so that KARE anchor Mike Pomeranz even poked fun at it. The conservative approach is also reflected in the station’s decision to have an extended outlook that’s two days shorter than other weather outlets, although they seem to stray from that when weekends are at the end of the horizon.

KARE recognized the possibility of accumulating snow last Tuesday, leaving us to wonder what they would have forecast had they been able to make a call on the storm last Monday, when it was already on the radar of others. On Wednesday night, the time that seems to have separated the men from the boys on this forecast, KARE foresaw the heaviest snow falling south of I-90. By Friday evening, when all others were saying 5-10 and 6-12, KARE continued with a guarded prediction, telling us, “Over half a foot of snow not looking all that improbable.”

No tweets emanated from @KARE11wx (Twitter) after Friday morning, which is hard to understand. We were also disappointed with the KARE’s WeatherNow cable segment, which was often a step slow, yet is portrayed as a timely weather source. C

Star Tribune
It’s clear to us that weather bloggers have an advantage over their television brethren, at least those who choose not to use their station’s website as a means for explaining things in more detail (a no-brainer to us). The Star Tribune blog update is typically prepared after the 10 p.m. news shows and often works with more recent data, so it’s generally a better source of timely information.

The Strib’s performance was hot and cold, echoing its own reference to the “on-again, off-again” storm. The Strib properly recognized the storm’s potential last Monday. And while they never completely let go of the idea – suggesting that “we’re still not out of the woods yet” – there was a statement that the “storm appeared to be fizzling.” By Wednesday afternoon, the Strib declared “eye-opening” changes, beginning to see the potential for a plowable storm, but called a foot “doubtful.”

As the storm moved closer on Thursday and Friday, the Strib anticipated the heaviest snow falling to the north of the metro, which did not happen. Beyond that, particularly on Saturday blog entries, there was information that seemed to contradict itself and confuse us.

Still, the Strib’s grade is buoyed by its commitment to keeping us informed on a timely basis, although its forecast was not top-notch this time around. B-

MPR
We don’t always include MPR’s weather blog in our grading because their blog is generally not updated on weekends. But since they were on top of the storm with solid coverage, we wanted to include them in our grades.

MPR identified the possibility of accumulating snow as far back as last Monday and reiterated the possibility on Wednesday. They were generally on target and always provided a solid plate of information for true weather junkies.

However, the one aspect of MPR’s forecasting performance that puzzled us was the two-and-a-half day lag between updates on snow prospects during a time other weather outlets were updating fast and furiously. If MPR is your chosen weather source, you had to be frustrated by the dearth of information at a time when you needed it most.

We also thought that MPR’s whopping range of 10-20 inches was too wide, at least for grading purposes (which may not count for much) of focusing on a single location. However, for the purpose of predicting metro-wide accumulations, it ended up being a fair reflection of the variance across the area. B

NWS
The NWS was very solid with this storm, identifying the possibility early on and maintaining a general steadiness of prediction. Their ultimate forecast of snow totals for the metro was right on and their range was very narrow (12-14 inches). As TMF didn't chronicle all the related NWS commentary (saving that, instead, for all the local forecasters), our review is brief. A
Agree? Disagree? Wondering why we wrote a master’s thesis? Let us know your thoughts.

19 comments:

  1. Bill, great analysis. I personally don't have as much time as I would like to analyze all the media outlets, so this is a great summation. Before finding your blog (last week) I used the NWS website for the bulk of my weather tracking. Since then I am now following your blog, MPR and Paul Douglas.
    Speaking of the NWS, I thought they did a fairly good job. They do give some technical insight with their "forecast discussion" which I wish they updated more frequently. Thanks for keeping me up late at night montioring all of this!

    ReplyDelete
  2. Bill where is your analysis of the NWS?

    ReplyDelete
  3. Bill, where's the running grade totals with average grades?

    ReplyDelete
  4. Wow, if Paul Douglas gets a B-, I want you to be my instructer for every class I ever take for the rest of my life!!! Maybe you can do my performance reviews at work too? Called off the storm completely, said a foot looks doubtful, said heaviest go to the north and was wrong, wrong, and wrong in a HUGE way and he gets a passing grade? Are you two related or something?

    ReplyDelete
  5. I totally agree with your comments about KARE. It's so frustrating to listen to their forecasts. I'm a loyal Paul Douglas reader because I do enjoy actually looking at the various models, which he posts on his blog for us to see.

    Thanks for doing this Bill.. it's pretty awesome!

    ReplyDelete
  6. I am a federal meteorologist - I can try and answer questions through the day on how the NWS operates if anyone has any.

    As for the technical discussion - called the Area Forecast Discussion or AFD - they are updated at least eight times daily: four to discuss the overall forecast (4AM, 10AM, 4PM, 10PM, plus or minus an hour); four to discuss the aviation forecast for MSP and other nearby airports (6AM, 12PM, 6PM, 12AM). Sometimes the AFD is updated more frequently if there are any other changes that want to be communicated.

    -P

    ReplyDelete
  7. I agree with the KARE grade. I watch, and like the news portion for their broadcast. I don't like how they conduct their weather segment. I find myself going to other outlets to find the detailed information I want.

    I do agree with Paul D's grade. I think it was fair considering he was monitoring the storm daily well before others were mentioning it. He shouldn't be dinged that much for relaying what the models were saying at that given time. And for that matter he wasn't trying to be an alarmist.

    ReplyDelete
  8. @Big Daddy, Our self-imposed deadline didn't allow enough time to grade the NWS. We'll work on it.
    @Annonymous, Follow this link for the cumulative grades: http://www.minnesotaforecaster.com/p/report-cards-to-date.html. Note that it's not fully up to date.

    Thanks for the feedback, all!

    ReplyDelete
  9. @federal meteorologist Thanks for your valued input! Feel free to email us at mnforecaster@gmail.com in the future if you have improvement suggestions.

    ReplyDelete
  10. Re: the KARE11 weather treatment... We're hoping to line up a visit with Sven Sundgaard to get a better understanding of their forecasting philosophy.

    ReplyDelete
  11. I'm an avid reader of the AFD's from the WFO's in the region.... MPX, FSD, ARX and sometimes other WFO's during extreme events, interesting reading.

    "Anonymous said...

    I am a federal meteorologist - I can try and answer questions through the day on how the NWS operates if anyone has any.

    As for the technical discussion - called the Area Forecast Discussion or AFD - they are updated at least eight times daily: four to discuss the overall forecast (4AM, 10AM, 4PM, 10PM, plus or minus an hour); four to discuss the aviation forecast for MSP and other nearby airports (6AM, 12PM, 6PM, 12AM). Sometimes the AFD is updated more frequently if there are any other changes that want to be communicated.

    -P"

    ReplyDelete
  12. If I might add.....hope that 2/28 storm trends a bit west/NW. Could give us a good storm.

    ReplyDelete
  13. most outlets are now commenting that they are watching the Sun-Monday System. Best explanation this morning that I saw is at KARE..click on weather, than forecast, they have a nice right up on it, and show it in the webcast.

    ReplyDelete
  14. I am glad to hear that PD often blogs on potentials that usually never come true - I sure hope his long range temperature comments aren't going to come true - staying below freezing through the middle of March - the flood risk is really getting worse and below average cold followed by a sudden onset of spring, as opposed to a slower gradual transition, is not good. Apparently the jet stream is showing signs of sticking around and staying cold. I have heard they are warning people living along metro creeks - Minnehaha, Purgatory, Nine Mile, etc., not just the bigger rivers in the region. My prayers are already with those that are going to endure the stress of this.

    As for personal (and growing collective) opinion, winter is just getting really, really old. There are far more people than not that simply don't care for winter, period, but this year is ridiculous. "Snow rage" is starting to manifest - people are depressed, irritable, and just plain on edge. We all need spring! Away with the snow, away with the jet stream to the north, and away with the below freezing temperatures. We are done with it, but I'm not so sure it wants to give up - it's looking like it might not be done with us. Boo hoo.

    ReplyDelete
  15. Randyinchampin...glad someone else brought that up, I see as well most of the area forcasters are "wataching" a storm on Sunday-Monday that if it slides any further Northwest could leave us in the bullseye. I must say though, the way the GFS looks im suprised their not saying more about it, it looks closer to us than they're making it out to be. I'd defintly start to watch this closly.

    I might also add, PD has posted on his blog yesturday and the day before almost an inch of liqid precipitation is being printed out from his models around March 6-9. If that turns out to be something, I think Paul should get some credit...a little suprised Dahl hasn't mentioned anything about it, he seems to act quickly with long range things as well...hhmm.

    ReplyDelete
  16. It seems as though PD relies on the GFS while Dahl looks at the EC model. Also if you add Tom Skilling as a friend on facebook he does a nice job posting snapshots of both models on fb. Although he deals with chicago weather, a lot of the maps include MN.

    ReplyDelete
  17. Well Dave D probably relies on the Euro because it's the "Champ" this winter.....rather than the 'Good For Show' GFS.

    Really, I'm sure KSTP has their own in-house model, plus they also use the globals.

    ReplyDelete
  18. actually i think the gfs outpreformed the euro this last storm. Euro was always the furtherest south with the nam way north and the gfs right in the middle.

    ReplyDelete
  19. PD mentions nothing about sunday monday....wcco has it staying south...kare11 is non commital about the track as is kstp...but the caveat is that it could easily swing further north giving us many inches..I love this site BTW

    ReplyDelete