Putting the Rain-Predicting Apps to the Test

RainAware Reigns Supreme in Precip-Predicting App Market

The Background

Two things are abundantly clear at this point in civilization. One, weather technology is becoming ever more complex and sophisticated, capable of detecting things once thought impossible. And two, man is always seeking to gain some measure of “life management” over the whims of Mother Nature. And so it seems only natural that new precipitation-predicting weather apps have emerged on the scene, ready to guide us through the day without getting wet.

The Test

We decided to give RainAware, Dark Sky and Ourcast, three of the newer rain-predicting apps, a test on a stormy night in Minnesota. (See also our exclusive interviews with the founders of RainAware, Dark Sky and Ourcast.) We began checking each app beginning at 6:45 p.m. and subsequently recorded their predictions every 15 minutes thereafter until the rain began. Likewise, once it became clear the rain would eventually end, we recorded the apps’ predictions for rain-ending times starting at 12:45 a.m., and then rechecked the apps every 15 minutes until the rain ended.

The Results

As our results in the accompanying graphic reflect, RainAware was the most accurate in determining both the beginning and ending times of the rain. RainAware locked on to the precipitation early and rather accurately. It came quite close to predicting the actual time of rain onset a full hour and a half before it arrived. And while it initially waffled a bit on the actual start time and experienced a server problem that made updates inconsistent for a short period, it provided a rain starting time nearly three hours in advance. In contrast, Ourcast seemed to think it was raining a full two hours before a single drop fell from the sky. Dark Sky, which doesn’t predict rain until it sees its arrival within a one-hour window from the current time, was slow to pick up on the ultimate arrival of the rain. At 8:45, Dark Sky predicted rain would begin at 9:35, when in fact it began at 9:15.

An analysis of predicted starting and stopping times revealed that RainAware was the most accurate.

RainAware was equally impressive in predicting an accurate time for the end of the rain. At 12:45 a.m, RainAware predicted the rain would end at 1:23. Dark Sky predicted the rain to last through 1:45 and Ourcast predicted the precipitation to last through at least 2:10 a.m. The rain ended at our location at 1:20.

Ground Clutter a Challenge for Dark Sky and Ourcast

Both Dark Sky and Ourcast also had challenges grasping the ultimate end of the rain. Both apps – to varying degrees – continued to think it was raining after the rain had actually stopped. The inability to decipher ground clutter from precipitation appears to be a continuing problem for both Dark Sky and Ourcast, as we’re seeing a reoccurrence of the problem as of this writing (May 2, 9:50 p.m.). While not perfect, it’s clear to us that RainAware is the superior app when it comes to detecting real rain from radar noise.

RainAware

In addition to RainAware’s actual performance in predicting rainfall, we also think the app’s features are generally the best of the apps tested. RainAware provides the longest lead time in rain prediction with a three-hour window. The three-hour window “messages” also come with informative statements about possible rain events even when there are no specific rain times. For example, it will suggest “showers could develop at any time,” or “dry now but a growing chance of rain” that we think provide a valuable “heads up” to users.

We also like the very simple but effective 7-day weather forecast that RainAware includes. While the main purpose of the app is to provide start and stop times for precipitation, the big-picture forecast means there’s no need to consult other apps for more general weather information.

Users desiring a pretty or interactive radar may be disappointed by RainAware. However, we think the radar is far secondary to the main function of the app, which is to provide start and stop times for precip. Besides, there are a number of other apps on the market dedicated exclusively to radar.

Dark Sky

Dark Sky brings undeniable beauty to radar depictions, which historically have been clunky and jittery. We also appreciate that all the information is boiled down to one screen, which includes confidence and forecast of precipitation strength. The app also provides the ability to backtrack two hours on the radar so that one can see what amount of precipitation passed through the area. Clearly, there’s some good innovation at work in this app.

However, we think the one-hour forecast window is insufficient, particularly when there’s no other information related to the overall forecast. If it’s noon and you’re wondering about the odds of getting in an evening softball game, Dark Sky is not going to help you.

Ourcast

The feature we liked best about Ourcast, the only free app among the three we tested, was the ability to move quickly and smoothly from one point on the map to another. This functionality is not present in Dark Sky or RainAware. Also, if you’re a fan of being social with your weather, Ourcast provides the opportunity to commiserate with your neighbors. Otherwise, we weren’t particularly impressed by Ourcast.

Conclusion

For our money, based on both the results of our test and its overall features, RainAware is the best precip-predicting app on the market.

The Minnesota Forecaster provides analysis of both the weather and those who forecast it. For periodic updates, follow us on Twitter and Facebook.

Feature App Profile: Dark Sky

Dark Sky is one of several new apps on the market that seek to provide specific times for the arrival and departure of precipitation (we previously featured RainAware). We sought to learn more about the app and had the following Q&A with Adam Grossman, co-founder of Dark Sky.

What are the differences between the iPad and iPhone versions?

They both have the same data, but the interfaces are different to reflect the difference in form-factor. On the iPad, for example, we show the radar map and future prediction on the same screen, whereas we split them up on the iPhone. Personally, I like exploring the radar on the iPad's big screen and use the iPhone to check the weather on the go.

Were meteorologists included in the development of the app?

Actually, no. My schooling was in physics, not meteorology, and the other two co-founders are both computer guys. The whole thing started as a side project a couple years back, just to see if something like this were possible. I got sick of getting stuck out in the rain, and decided to experiment with applying statistics and machine learning to the problem (since I lacked traditional meteorology experience). Against all odds, it seemed to work really well for prediction precipitation in the near term.

Is weather knowledge built in to the app or is it more an interpretation of the movement of cells on the radar?

We take a statistical approach, rather than using physical/meteorological models. So the predictions we make are based on how the particular storm -- and storms like it -- have moved and developed in the past.

What do you see as the advantage of your radar depiction (vs. traditional radar)?

For a lot of people -- especially those who aren't weather nerds and aren't used to looking at and interpreting weather radar -- conventional weather animations out there can be very confusing. Because the doppler radar stations only take new images every five to ten minutes, they tend to result in clunky, jerky animations that are hard to follow. So what we've done is take our prediction algorithms and apply them to the time periods in between the radar frames, allowing us to create smooth and fluid animations. It really makes it a lot easier to see how the storms are moving and changing, and where they're headed.

Have you tested the accuracy?

An important part of our statistical approach is that the system constantly monitors its own accuracy: Every time a new radar images comes in, we use it to compute the error of past predictions. Because of this, we can tell in real-time which storms we have accurate predictions for, and which ones we don't. We reflect this in the interface as a "wobble" in our graphs. The more wobble, the less confident in our predictions we are.

Are users more apt to be overwarned or underwarned about precip?

It depends on the area and the type of precipitation. Light, spotty, slow moving precipitation is the hardest for us to predict, so in those conditions you might get some light sprinkling that we didn't anticipate. On the other hand, radar images also have a lot of "noise" in them (i.e. regions that look like precipitation that actually aren't) so if we don't do a perfect job cleaning them up, it can lead to false-positives. Fortunately, as we gather more data, we're constantly improving both our cleaning and prediction algorithms.

Are we correct in assuming the window of projection is one hour?

Right now we're restricting it to a forecast for the next hour. We plan to expand beyond the hour in future app releases, as we improve our prediction capabilities.

Does the radar show the past hour of precip as well?

Yes. You can scrub back in time over the past 2 to 3 hours. On the iPad, there's a history button to load this past data (which will be clearer in an update of the app we have coming out in the next day or two).

Are there locations where the performance of Dark Sky is less reliable or precipitation patterns that are more problematic?

We're less reliable in places with off-and-on sporadic light rain that just sort of sits over an area all day: think Seattle. We're most accurate for stronger storms, such as thunderstorms and those nice cohesive squall lines you'll often see rolling down the plains. 

What do you consider the strengths of your app?

Our goal was to make a weather app that was easy and fun to use for everyone, not just weather junkies. So we've put an emphasis on design, usability, and making the best radar visualization out there. I think that really sets us apart from the other apps out there.

  

Where do you see the most opportunities for improvement in future upgrades?

We have a big list of improvements we want to make, some minor and some major. One of our biggest priorities is a notification system: The app is only helpful if you remember to consult it, and I've personally been caught off guard by the rain because I just didn't think to check the app. So we want to build in notifications that will actively warn people when rain is headed their way.

Improvements to the underlying prediction algorithm are also a huge priority, of course. We're constantly improving things behind the scenes, and users should get the benefit of those improvements even without having to update the app.

RainAware App: Rain Predictions to the Minute?

Scanning the wire recently, we learned of a new weather app that seemed both unique and practical. The goal of RainAware, currently available as an iPhone app, is to “predict to within minutes when precipitation will reach a user’s exact location, up to three hours in advance.” Knowing that weather can be challenging to predict, we were curious to learn more about this app and how it works (and no, we’ve not yet tried it).

We interviewed Ryan McGee of RainAware to learn more about the app and its capabilities. Here’s what he had to say:

How exact is “exact?”

We try to be precise to within half a mile and three minutes. This would be considered a “perfect” forecast. Compare this to any other forecast, which is typically for a general area AND only to within an hour or two. There are several limiting factors to how precise we can get:

1) Pixel resolution of radar images and radar bin size. We go down to one kilometer resolution here when locking onto and moving areas of precipitation.

2) Radar data frequency. Typically 5 to 10 minutes, depending on what VCP (volume coverage pattern) the radar is in. The program looks for new information every minute and recalculates once new data is available, typically every five minutes.

3) The DBZ value used for precip. This can change depending on precipitation type, distance from the radar and vertical humidity profiles. These are really just estimates since the values vary. This could add a couple minutes of uncertainty.

4) Precipitation motion. This is a key element, and is difficult to attain accurately. We use several methods for extracting this information but our cornerstone is the “ensemble” method, which uses thousands of slightly different motion vectors and then uses probabilities to convert to most likely rain times.

How confident are you that future radar can be predicted?

We do not actually produce a “future” radar with our system, but we know there are some who try. This may be possible in some situations, but in other situations like summertime pulse storms it is impossible past a few minutes.

What kind of testing have you done? Has accuracy been assessed?

We have tested it thousands of times over the past two plus years, through all seasons and all areas of the country. With each case we observed, if it did not perform well, we investigated and re-wrote the code. In fact, this process will always be ongoing, in order to keep improving upon the system.

It is very difficult to make a “one size fits all” scheme, so we offer user settings that can affect output. Namely, the precipitation threshold. If users, for whatever reason, seem to get too many false alarms, they can raise the criteria. The converse is true for lack of detection.

Are there particular geographic locations where you’ve tested the app?

We have tested it everywhere. There are gaps in radar coverage over the mountains of the western U.S., so it can’t work there. Also, it helps to have multiple radars covering the same area, which is true of the central and eastern U.S. That way if a radar is down you have a backup. There is logic build into the app that goes into choosing which radar is used. Then there are some areas that may have clutter more often than others. While we do employ clutter suppression, sometimes this comes through, especially near mountains.  Lake effect snow is another area we will be improving upon by next winter.

How much development has gone into the product?

Over two years of testing and development and thousands of hours of coding. We are perfectionists, so this will be ongoing.

Are users more apt to be over-warned or under-warned?

We want to err slightly on the side of detection, but this is not a large bias. And again, users can adjust to one of three different sensitivity levels to suit their needs.

If the app says it will rain in 13 minutes what are the odds that that will actually happen?

At thirteen minutes out, it is highly likely it will rain but allow a few minutes (maybe three or so) buffer to account for the errors previously discussed. We use an ensemble of motions and the output is probability based so that “13 minutes” is based on a probabilistic threshold. If you look at the bar graph (third page of the app’s main screen), you will be able to see the probabilities (bar heights). The red horizontal line is the “best” threshold to use based on testing, and optimizing CSI [critical success index] scores for rain (CSI is a bit POD [probability of detection] biased). So while it might say, “It’s gonna rain in 13 minutes” on the main timer screen, a quick swipe will allow you to see how confident we are that it will occur. We feel the default threshold, out of the box, exhibits the best all around skill level.

What about instances in which rain develops over top of a location, essentially without warning?

In these cases, rain times will only appear once precipitation is detected on radar. We do not claim to be able to work magic, but we do claim to lock onto precipitation as fast as humanly possible. With 1-minute checks for new data, we are sure to do that. Now, we DO have some special wording in situations where there is a high likelihood of “sudden” development. We may say, “Dry for now, but good chance of precipitation later this afternoon,” or we may even say, “Precipitation may develop at any time!” Further, if there are showers somewhat upstream of the user, but not necessarily “going” to hit them, we may mention there are showers in the area. Summertime “pop up” storms is another area we will be focusing on in the near future. This is one of the hardest weather elements to try to “put in a box.”

Are there circumstances you expect will be better handled than others? (i.e., winter rain seems more predictable than summer pop-up storms)

You are correct. Summer pop up storms in weak flow environments are hard to lock onto and properly move. But the question is whether any human can do better staring at a radar full of pulse storms popping up. Our goal is to be as good or better than a human meteorologist. We do have methods we are working on to optimize this, so stay tuned.

And yes, winter precipitation is more easily handled because of good flow aloft/steering currents. But light snow can also be hard to detect, especially if you are not close to the radar. It is difficult to determine whether you are simply seeing clouds and virga, or if there is light precipitation occurring.

Very light precip, such as drizzle or sporadic flurries, may not be handled well, just due to radar detecting capabilities. That said, we have seen RainAware pick up shallow heavy drizzle and mist on days when the official forecast simply said “Mostly Cloudy.”

Do you have an idea of future upgrades and additional capabilities consumers can expect down the road?

One of our first updates will include the ability to choose from a list of about 1,000 cities across the U.S. Currently, the program is limited to one’s GPS location and a short list of sites. Look for this in version 1.0.1, coming in the next few weeks.

We understand the radar imagery in our app is basic, as it is meant to be more of a supplement rather than an all-out, full tilts program. We could spend time on improving the radar, such as adding pinch zoom. However, we are trying to focus on perfecting rain start and stop times currently. We are aware of its limitations.

We expect the user interface may be redesigned at some point as well. Version 1.0.0 is only the beginning. We have a place on our website for users to enter feedback and comments that will help improve the system.

Ultimately, RainAware has one main goal: to tell the user what time precipitation will begin and end. Our vision is to someday make this information as easy to get as the time of day. Imagine always having a rain time wherever you see a clock. That’s an awful lot of computing power, but possible.

Have you tried RainAware? Feel free to leave your comments.