Wednesday, May 2, 2012

Putting the Rain-Predicting Apps to the Test


RainAware Reigns Supreme in Precip-Predicting App Market

The Background
Two things are abundantly clear at this point in civilization. One, weather technology is becoming ever more complex and sophisticated, capable of detecting things once thought impossible. And two, man is always seeking to gain some measure of “life management” over the whims of Mother Nature. And so it seems only natural that new precipitation-predicting weather apps have emerged on the scene, ready to guide us through the day without getting wet.

The Test
We decided to give RainAware, Dark Sky and Ourcast, three of the newer rain-predicting apps, a test on a stormy night in Minnesota. (See also our exclusive interviews with the founders of RainAware, Dark Sky and Ourcast.) We began checking each app beginning at 6:45 p.m. and subsequently recorded their predictions every 15 minutes thereafter until the rain began. Likewise, once it became clear the rain would eventually end, we recorded the apps’ predictions for rain-ending times starting at 12:45 a.m., and then rechecked the apps every 15 minutes until the rain ended.

The Results
As our results in the accompanying graphic reflect, RainAware was the most accurate in determining both the beginning and ending times of the rain. RainAware locked on to the precipitation early and rather accurately. It came quite close to predicting the actual time of rain onset a full hour and a half before it arrived. And while it initially waffled a bit on the actual start time and experienced a server problem that made updates inconsistent for a short period, it provided a rain starting time nearly three hours in advance. In contrast, Ourcast seemed to think it was raining a full two hours before a single drop fell from the sky. Dark Sky, which doesn’t predict rain until it sees its arrival within a one-hour window from the current time, was slow to pick up on the ultimate arrival of the rain. At 8:45, Dark Sky predicted rain would begin at 9:35, when in fact it began at 9:15.

An analysis of predicted starting and stopping times revealed that RainAware was the most accurate.
RainAware was equally impressive in predicting an accurate time for the end of the rain. At 12:45 a.m, RainAware predicted the rain would end at 1:23. Dark Sky predicted the rain to last through 1:45 and Ourcast predicted the precipitation to last through at least 2:10 a.m. The rain ended at our location at 1:20.

Ground Clutter a Challenge for Dark Sky and Ourcast
Both Dark Sky and Ourcast also had challenges grasping the ultimate end of the rain. Both apps – to varying degrees – continued to think it was raining after the rain had actually stopped. The inability to decipher ground clutter from precipitation appears to be a continuing problem for both Dark Sky and Ourcast, as we’re seeing a reoccurrence of the problem as of this writing (May 2, 9:50 p.m.). While not perfect, it’s clear to us that RainAware is the superior app when it comes to detecting real rain from radar noise.

RainAware
In addition to RainAware’s actual performance in predicting rainfall, we also think the app’s features are generally the best of the apps tested. RainAware provides the longest lead time in rain prediction with a three-hour window. The three-hour window “messages” also come with informative statements about possible rain events even when there are no specific rain times. For example, it will suggest “showers could develop at any time,” or “dry now but a growing chance of rain” that we think provide a valuable “heads up” to users.

We also like the very simple but effective 7-day weather forecast that RainAware includes. While the main purpose of the app is to provide start and stop times for precipitation, the big-picture forecast means there’s no need to consult other apps for more general weather information.

Users desiring a pretty or interactive radar may be disappointed by RainAware. However, we think the radar is far secondary to the main function of the app, which is to provide start and stop times for precip. Besides, there are a number of other apps on the market dedicated exclusively to radar.

Dark Sky
Dark Sky brings undeniable beauty to radar depictions, which historically have been clunky and jittery. We also appreciate that all the information is boiled down to one screen, which includes confidence and forecast of precipitation strength. The app also provides the ability to backtrack two hours on the radar so that one can see what amount of precipitation passed through the area. Clearly, there’s some good innovation at work in this app.

However, we think the one-hour forecast window is insufficient, particularly when there’s no other information related to the overall forecast. If it’s noon and you’re wondering about the odds of getting in an evening softball game, Dark Sky is not going to help you.

Ourcast
The feature we liked best about Ourcast, the only free app among the three we tested, was the ability to move quickly and smoothly from one point on the map to another. This functionality is not present in Dark Sky or RainAware. Also, if you’re a fan of being social with your weather, Ourcast provides the opportunity to commiserate with your neighbors. Otherwise, we weren’t particularly impressed by Ourcast.

Conclusion
For our money, based on both the results of our test and its overall features, RainAware is the best precip-predicting app on the market.

The Minnesota Forecaster provides analysis of both the weather and those who forecast it. For periodic updates, follow us on Twitter and Facebook.

16 comments:

  1. Bill, this is great stuff and thanks for taking the time and effort to put all of this together.

    What I find most interesting is that "human" predicted almost as well as the Rainware app. and better than the other 2 apps.

    I'm not surprised.

    ReplyDelete
  2. Check that, "human" appeared to predict BETTER than all 3 of the apps.

    ReplyDelete
  3. Actually, RainAware beat the human almost every period from 7:30 through rain time (9:15).

    ReplyDelete
  4. 2+ hours before it actually started raining, 'Human' performed much better (20 min. +) than RainAware when predicting the start time. After that period, it was basically a wash.

    For me, it is more impressive when a forecast can accurately predict the start of precipitation well in advance (1 hour +) rather than right before the echoes are on your doorstep. With that being said, I believe that 'Human' was the winner in this test.

    ReplyDelete
  5. A bit more info on this and the “human” involved. The human was me, a weather enthusiast but not an informed meteorologist. So I’m not overly sophisticated about these things, though I can read a radar. Also, I used a Weather Underground radar screen and am not able to see my precise (to the street intersection level) location. With access to more detailed radar, I may have been able to do a little better. However, the apps were using my IP address and I don’t know exactly how precise that gets, either. (Presumably, the apps perform best when using GPS.)

    As for rain ending time, which was tested in a less rigorous manner, I can report anecdotally that I’ve been a little better in noting the faster movement of the precip and therefore rain-ending times. That may have to do with the character of the storms involved, and I’m told that RA is working to tweak that aspect. Still, however, compared to the other apps, RA was superior.

    Tom, it would be interesting for you to compete in the contest some time. Then we could have a “human meteorologist” as a contestant. My thinking is that if these apps can do close to as well as a human meteorologist and don’t get messed up with ground clutter (a tall order!), then there is real value, particularly to someone who doesn’t have access to their own personal meteorologist.

    As an aside, I just checked into The Weather Channel’s Trupoint feature which is supposed to predict beginning and ending times for rain. It’s horrible, and even though it’s in Beta, I don’t understand why they’d bring forth a product that’s clearly not ready for prime time.

    ReplyDelete
  6. randyinchamplinMay 7, 2012 at 9:30 PM

    Time for me to reply. 1st I don't have a smart phone, but if i did I think it would be a waist to just have a app to tell me when it was going to rain. Rain (if not heavy enough to cause flash flooding) will not threaten my life. Give me a radar app that shows base reflectivity, and base velocities and I would be fine, but that's just me, I can read some radar scans.

    But having said that, most of the public can't do that, and are not willing to learn it. That is cool, and I understand that. What I'm trying to say is give me a app that shows where lightening is occurring, that I would pay for, because if rain is moving in with a electrical storm that would threaten my life. Now that I would pay for.

    ReplyDelete
    Replies
    1. Randy, I think the primary audience for these apps is not weather enthusiasts but for people that aren't knowledgeable about weather. I think of friends in the neighborhood who call me to ask if they can get a walk in before the rain arrives... and if one of these apps were in their hand, would it be an adequate replacement for such a service.

      Delete
    2. randyinchamplinMay 8, 2012 at 7:54 PM

      Bill, point well taken

      Delete
  7. Bill, I had an idea for your next poll. Simple question. Do you own a weather radio?

    ReplyDelete
  8. When RainAware has the storm motion pegged...you simply cannot beat it because it's an exact calculation. It's only when the vector is off by 10 or 20 degrees that the human can beat it. This is the trickiest part of it all (Clutter can be tricky too but we have that problem 95% solved). There are some major improvements that will occur over the coming months.

    ReplyDelete
  9. I'm posting this here since it is the most recent thread...Weather has been slow as of late, but signs of an active pattern may be shaping up to end next week and into the Memorial Day weekend. Something to keep in mind as many may be out camping and away from their homes. Get a weather radio, and bring it with you. Even a little portable TV wouldn't be a bad idea with some batteries in it. At this point in time though, Memorial weekend could end up being pretty hot, even hotter than today (Fri) or tomorrow. Increased heat and humidity can spell trouble on the storm front, but it may end up being where it is just a big dome of hot air, and everything remaining capped or riding the ridge up to our north into Canada. Enjoy the slight cooldown early next week because we may end up getting into a much hotter, humid, and overall typical summer pattern come the end of next week.

    ReplyDelete
  10. Just curious, how did the rain apps do today? Rain started just like I thought it would for the region, some where between 5-7pm for the metro. The next question I have is this. Do these apps also predict the end of the rainfall? I will predict here in the metro that we see are last of the rains tomorrow (Sunday 5/20) around 5pm.

    ReplyDelete
    Replies
    1. Didn't test the rain apps out today because I'm in California. The apps do predict the end of the rainfall but only within the narrow window in which they operate (which ranges between one hour for Dark Sky to three hours for RainAware).

      Delete
    2. You missed a pretty good storm, especially in St Paul. Lots of hail.

      Delete
  11. hi there the weather pre-perdidting apps will get more and more acurate with update lets check back in in retest next year the updates shoukd be out

    ReplyDelete
  12. Nice weather prediction app, interesting. It will work. Great idea admin. I like it. keep it up !!! Visit college paper for best papers.

    ReplyDelete