Tuesday, May 22, 2012

Considerable Forecaster Variability for Mem Day Weekend

Analysis: The Memorial Day Weekend forecast proved to be a challenge. The official high temperatures at MSP for Saturday, Sunday and Monday were 64, 92 and 80. None of the forecasters shined, though we'd say the Strib probably did the best, coming very close to nailing both Sunday and Monday temps. WCCO was out to lunch with the Saturday forecast, missing it by 15 degrees and standing out from the crowd in the process. This just in: long-range forecasts, particularly for this part of the country, is not quite an exact science!

A changeable forecast for Memorial Day Weekend means great variability in forecasts from your beloved local and national weather outlets. For Saturday, Sunday and Monday, there are spreads of 7 degrees, 14 degrees and 9 degrees respectively between forecasters. In general, the Star Tribune is the most bullish on weekend warmth (followed closely by KSTP) while WCCO is generally the coolest. Here are temperature forecasts as obtained at noon on Tuesday.

#4: 76/77/72
#5: 74/86/82
#9: 74/81/73
#11: 75/na/na
NWS: 74/83/73
Accuweather: 70/81/70
TWC: 69/85/72
Strib: 72/91/81

Wednesday, May 2, 2012

Putting the Rain-Predicting Apps to the Test

RainAware Reigns Supreme in Precip-Predicting App Market

The Background
Two things are abundantly clear at this point in civilization. One, weather technology is becoming ever more complex and sophisticated, capable of detecting things once thought impossible. And two, man is always seeking to gain some measure of “life management” over the whims of Mother Nature. And so it seems only natural that new precipitation-predicting weather apps have emerged on the scene, ready to guide us through the day without getting wet.

The Test
We decided to give RainAware, Dark Sky and Ourcast, three of the newer rain-predicting apps, a test on a stormy night in Minnesota. (See also our exclusive interviews with the founders of RainAware, Dark Sky and Ourcast.) We began checking each app beginning at 6:45 p.m. and subsequently recorded their predictions every 15 minutes thereafter until the rain began. Likewise, once it became clear the rain would eventually end, we recorded the apps’ predictions for rain-ending times starting at 12:45 a.m., and then rechecked the apps every 15 minutes until the rain ended.

The Results
As our results in the accompanying graphic reflect, RainAware was the most accurate in determining both the beginning and ending times of the rain. RainAware locked on to the precipitation early and rather accurately. It came quite close to predicting the actual time of rain onset a full hour and a half before it arrived. And while it initially waffled a bit on the actual start time and experienced a server problem that made updates inconsistent for a short period, it provided a rain starting time nearly three hours in advance. In contrast, Ourcast seemed to think it was raining a full two hours before a single drop fell from the sky. Dark Sky, which doesn’t predict rain until it sees its arrival within a one-hour window from the current time, was slow to pick up on the ultimate arrival of the rain. At 8:45, Dark Sky predicted rain would begin at 9:35, when in fact it began at 9:15.

An analysis of predicted starting and stopping times revealed that RainAware was the most accurate.
RainAware was equally impressive in predicting an accurate time for the end of the rain. At 12:45 a.m, RainAware predicted the rain would end at 1:23. Dark Sky predicted the rain to last through 1:45 and Ourcast predicted the precipitation to last through at least 2:10 a.m. The rain ended at our location at 1:20.

Ground Clutter a Challenge for Dark Sky and Ourcast
Both Dark Sky and Ourcast also had challenges grasping the ultimate end of the rain. Both apps – to varying degrees – continued to think it was raining after the rain had actually stopped. The inability to decipher ground clutter from precipitation appears to be a continuing problem for both Dark Sky and Ourcast, as we’re seeing a reoccurrence of the problem as of this writing (May 2, 9:50 p.m.). While not perfect, it’s clear to us that RainAware is the superior app when it comes to detecting real rain from radar noise.

In addition to RainAware’s actual performance in predicting rainfall, we also think the app’s features are generally the best of the apps tested. RainAware provides the longest lead time in rain prediction with a three-hour window. The three-hour window “messages” also come with informative statements about possible rain events even when there are no specific rain times. For example, it will suggest “showers could develop at any time,” or “dry now but a growing chance of rain” that we think provide a valuable “heads up” to users.

We also like the very simple but effective 7-day weather forecast that RainAware includes. While the main purpose of the app is to provide start and stop times for precipitation, the big-picture forecast means there’s no need to consult other apps for more general weather information.

Users desiring a pretty or interactive radar may be disappointed by RainAware. However, we think the radar is far secondary to the main function of the app, which is to provide start and stop times for precip. Besides, there are a number of other apps on the market dedicated exclusively to radar.

Dark Sky
Dark Sky brings undeniable beauty to radar depictions, which historically have been clunky and jittery. We also appreciate that all the information is boiled down to one screen, which includes confidence and forecast of precipitation strength. The app also provides the ability to backtrack two hours on the radar so that one can see what amount of precipitation passed through the area. Clearly, there’s some good innovation at work in this app.

However, we think the one-hour forecast window is insufficient, particularly when there’s no other information related to the overall forecast. If it’s noon and you’re wondering about the odds of getting in an evening softball game, Dark Sky is not going to help you.

The feature we liked best about Ourcast, the only free app among the three we tested, was the ability to move quickly and smoothly from one point on the map to another. This functionality is not present in Dark Sky or RainAware. Also, if you’re a fan of being social with your weather, Ourcast provides the opportunity to commiserate with your neighbors. Otherwise, we weren’t particularly impressed by Ourcast.

For our money, based on both the results of our test and its overall features, RainAware is the best precip-predicting app on the market.

The Minnesota Forecaster provides analysis of both the weather and those who forecast it. For periodic updates, follow us on Twitter and Facebook.

Tuesday, May 1, 2012

Feature App Profile: Ourcast

Ourcast is one of several new apps on the market that seek to provide specific times for the arrival and departure of precipitation (we previously featured RainAware and Dark Sky). We sought to learn more about the app and had the following Q&A with
Mark Hohmann, Ourcast founder.

We’ve read that the Ourcast app predicts rain and snow with "unprecedented accuracy." Can you tell me more about what’s been done to test and verify this accuracy?
The biggest challenge is in making an accurate correlation between what radar is showing and what is actually happening on the ground in terms of rain observations. We found that when you can correlate the past behavior of the radar with past rain observations, you get a much better indication of precipitation. Our accuracy refers to the ability to correlate the radar to a rain observation. It comes as a surprise to a lot of people that radar is a technical signal and while a good indicator, it’s not always directly correlated with what’s happening on the ground.

It sounds like statistical modeling is a major part of the app. How much meteorological knowledge was used in developing the app?
We found that if you’re going to predict the next two hours of precipitation, the dynamics associated with a statistical model are better than a meteorological approach which would include a numerical weather model or a mesoscale model.

Do you envision your app being used more by weather enthusiasts or common folk?
Both. We’re definitely focused on the typical consumer that’s not a weather enthusiast. We think that’s where the biggest gap is in peoples’ ability to use the weather forecast in the short term. We want to give those people that ability. But weather enthusiasts should find it useful as well as it can help them determine when radar is really describing precipitation on the ground.

Ourcast makes use of “check ins” as a component in the app. What do check-ins provide?
Check-ins are user reports about what is going on on the ground. The more information that users provide, the more we can build on our accuracy. Check-ins also provide a social aspect to the app by providing an ability to discuss weather with people near you.

How dependent on check ins are you for accuracy?
We launched with a product we think is accurate. It will get more accurate as our community grows. Check-ins and user reports help us to tease out what’s ground precipitation from what’s clutter, ultimately adding more accuracy.

The way that we correlate radar to precip is through a large set of weather stations located around the U.S. This includes backyard weather stations, government sources, proprietary sources and Weather Underground stations. This provides real-time information and gives more accuracy, almost to neighborhood level reports. These weather stations also provide more data beyond precipitation including temperature and wind speed.

Are users more apt to be overwarned or underwarned about precip?
We try to be as accurate as possible. However, we have built in a slight bias for overprediction. We think people would rather be warned more than warned less – they’d rather be warned of rain that doesn’t happen than rain that comes without being predicted. In other words, there’s a bias to false positives more than false negatives.

How far out does the forecast go?
In general, our forecast window is two hours. However, depending on when you check it, you may catch the system in a cycle and it can be a little less. But it should always provide at least a one hour and 40 minute forecast. Beyond two hours, the model right now degrades, and at that point mesoscale models and traditional numerical models become better.

Are there locations where the performance of Ourcast is less reliable or precipitation patterns that are more problematic?
There are regional issues such as where the radar signal is weakest or concentration of ground stations (for observations) is weakest. Mountainous regions have a lot of radar blockage and also tend to have fewer ground observations because of less population. There can also be issues with storm types. Storms that are strongly delineated, have a defined leading edge and are part of regional fronts usually provide the most reliable accuracy. Light snow in winter can be more of a challenge. However, as we grow our community it will mean a system that’s less reliant on radar and we’ll be able to build a forecast that’s more robust.

What do you consider the strengths or uniqueness of your app?
We’re unique because of a combination of a strong forecast system and a strong social app. We’ve designed it to have both. When you can make it more interactive, fun, social, etc., it takes on a new dimension. The social aspect can mean that even when plans are cancelled because of the weather, it can be a more fun, social experience.

Where do you see the most opportunities for improvement in future upgrades?
We want to keep making the app more interactive and fun. We want to add the concept of rankings and badges, so that users who report will get rewarded more. We are also working to provide an alert for users that will warn them ahead of time when precip is coming or when it’s about to end. We think that will really help to add value to the product.

In testing the app on an iPod Touch, we’ve had a few instances when “low memory” issues have caused the app to crash. Is that unusual?
No, it’s not and that’s because the app is not made to perform on the iPod Touch but rather on iPhones. However, it can still work on an iPod Touch but it may have some bugs until we support that platform more effectively.

The app categorizes rain into drizzle, light rain or heavy rain, but there’s no designation for moderate rain. Is that by design?
Yes, that’s by design. We want to keep things simple and just have a few separate categories. The app includes dBZ levels that reflect the intensity of precipitation, so people can see the see the relative changes in strength of precip over time.