Inside the Predict-o-Tron
The details of the Huffington Post’s NCAA forecaster
The HuffPost Predict-o-Tron is a tool we built to let people make their own March Madness bracket predictions using basketball statistics, expert ratings, and results from the past four tournaments.
There are some interesting tidbits to be found in the data, although they all need to be qualified with the understanding that model performance is based on only four years of data, which leaves us at risk of overfitting. This means that slider combinations that appear to do very well for the past four years may not continue to perform as well if expanded to the past 10 years. With that said, it looks like the experts are very good at picking a bracket, taller teams tend to do better than shorter teams, younger teams do better than older teams, and teams with more depth (both in scoring and playing time) do better than teams with less depth.
Designing the App
Our goal was to keep the interface as simple and easy to use as possible, but run a more sophisticated probabilistic bracket generator in the background.
The interface is made up of a set of 23 sliders (things like tournament seed, expert ratings, and offensive and defensive statistics), grouped into five categories, and the standard 68-team tournament bracket familiar to college basketball fans. The sliders allow users to adjust the weighting of each attribute either positively or negatively, and to see the effects of those changes as they make them. One thing that was important to us was to keep the bracket as vertically short as possible, so that the effects of different slider combinations can be seen by most users. The layout we ended up fits on a 13” screen. On narrower screens, such as mobile devices, we stack the regional brackets on top of one another; keeping the traditional bracket layout on a narrow screen would have made parts of it unreadable.
We decided to handle all the computation in the browser because we wanted users to see the effects of changes to their models as quickly as possible. Sending a request to a web server each time a slider was moved would have been impractical, as well: handling an unpredictable number of dynamic, uncachable requests was not something we wanted to deal with.
setTimeout() function also let us redraw the bracket before the previous years had finished updating.
D3.js is used in several areas. To create the sliders, we used the method described here. We also use d3 to bind our matchup data to the bracket, which meant we could update the bracket data in bulk, rather than manually looping through each game. (You can read more about d3 and data binding here.)
How It Works
Here’s how the bracket is populated:
- Generate the expected point spread between any two teams based on the positions of the sliders. Each of the individual attributes defines its own point spread between any two teams. The expert ratings imply point spreads, and the others are scaled linearly to between 1 and 30 points (a reasonable range given results and ratings from recent tournaments). We considered a few methods of scaling each attribute, but decided on linear scaling because it made the sliders behave most like a user expects. The total expected point spread between two teams is the weighted average of attribute point spreads based on their slider positions.
- Convert the total point spread between any two teams into probabilities of victory. To do this, we use the cumulative distribution functionwith a standard deviation of 11 (per Sokol and Carlin). These papers show that given a few years of historic NCAA basketball results, the probability that one team will beat another roughly follows a normal distribution with mean equal to the expected point spread, and standard deviation of 11.
- Using conditional probability, calculate the chance that each team will advance to each round based on a) the probability that they will reach the previous round, and b) the likelihood that they will beat each of their potential opponents (weighted by the chance they will face that opponent).
For fun, we also searched for the Predict-o-Tron model that shows the best results over the previous four years. Finding the best model is a challenge, since there are 201^23 (about ninety-four sexdecillion) possible slider combinations, and scoring a bracket has the added complication that early games affect later games. While it’s certainly overfit to the small number of tournaments, and it might not even be the Predict-o-Tron model that gives the best results over the past four years, we found one that would have correctly predicted 85 percent of games in the past four tournaments. We used simulated annealing to brute force our way toward a solution, and arrived at this model after a few days of processor time. Our approach was to swap back and forth between two phases: 1.) try many random models for a short period of time, then 2.) improve the best one by making small tweaks until we hit the local maximum and it couldn’t be improved any more.
The Predict-o-Tron could be enhanced with the addition of more years of data, other attributes that are historically good at predicting games, and home court advantages, but we chose to leave these out for simplicity and performance reasons.
We hope to create similar tools for other subjects in the future. Sporting events are a fairly obvious choice, but we see potential in other areas as well.