For most of the fall, some colleagues and I developed NYT 4th Down Bot, a news application that analyzed N.F.L. 4th down decisions in real time. I had been thinking about a project like this for a long time – Amanda Cox forwarded me David Romer’s paper at least three years ago – but the fall of 2013 was really the first time I felt like I was equipped to actually make a project like this happen.
Earlier in the year, while working on a graphic about the draft, I had spoken with Brian Burke, the creator of the popular football analytics web site Advanced NFL Stats. I explained my idea to him (roughly, a web site that analyzed every 4th down of every game) and asked him if he'd be willing to create an API for his 4th Down Calculator for a joint venture once the season started. He said yes, and I told him I'd be in touch in a few months.
The app turned out to be complicated. We're watching a live data feed of every NFL game (provided by a Canadian company called SportsDirect), processing the feed to determine the situation and result of the play, hitting an API representing Brian Burke's 4th Down calculator with that game situation, processing the response and publishing new data files to S3. There is no real backend, no database and no static content – just a bunch of Node scripts publishing JSON files every 15 seconds anytime an N.F.L. game is happening. (Not by design...it just happened that way. I'm not opinionated about these sorts of things.)
Start to finish, the whole process took about a minute. (Before Tom Giratikanon got involved, it took a lot longer than that – without him, the app would still be sitting on my Desktop.)
My first sketches were simply small tables designed for a phone using data from the 2012 season.
As the concept became clearer, the tables became bigger, more interesting, and much more opinionated.
I had been working on a demo for weeks before Shan Carter and Jen Daniel suggested the idea of personifying Brian's model. (Originally, the tables simply compared "Coach says" to "math says", clearly boring in retrospect.) This first started with "NYT Coachbot", a flat vector drawing from Jennifer.
We then started giving the robot more of a mouth.
A few sketches later, we were emphasizing the throwback style of 1980s computers and printers in the bot’s reports.
The more we designed, the more we realized most people would be coming to the "play" pages directly from Twitter, so we had to make them as clear as possible to the uninitated – this is why each robot decision comes with a dead simple sentence that says exactly what it is.
The drawings and renderings are from Jennifer Daniel and Shan Carter, who worked in a back-and-forth style, bringing the bot to life using Modo and Adobe Illustrator.
And what that looks like in Modo, the 3D program Shan used to render it.
Here are some of Jennifer's thoughts on reactions he could have. (Bummer about "ashamed" bot's lack of self-control.)
In the end, we didn't make all those poses, but we did do a lot of fun ones that really brought the concept to life. Here are all the things the 4th Down Bot liked to do (including dogging you for making nachos). It even fooled our friend Lynn Cherny when it signed off one night, which is almost as great as that time it was a copy editor on the foreign desk.
Here are all the poses Shan put the bot in at one point or another:
By the end of the season, it had more than 10,000 Twitter followers and caught the attention of various famous economists and CEOs of Twitter.
Still, it's certainly not perfect. Here are some things that are wrong with it:
It probably took too long to launch. We published around Week 13. Officially, we missed our goal by only five days, but in retrospect we could have published something less polished and improved it as it gained momentum. (This is hard for us newspaper people to do.)
It could feel more “live”. The lag between the end of the play and the analysis takes about a minute, but sometimes the delay on the play-by-play data lagged a bit, which meant you were getting bot analysis well after the other team started its drive. This isn't ideal, but there just wasn't much we could do about it.
Because it was programmed to analyze decisions that already happened, some aspects of N.F.L. play aren't captured well. For example, when a team intentionally takes a penalty on 4th and 1 near midfield, the bot applauds the punt on 4th and 6 without properly scolding the 4th and 1. This particularly annoyed Aaron Schatz of Football Outsiders, who later got over it.
As many statisticians noted, it could display uncertainty better than it does. From my perspective, that's the most legitimate criticism, and we hope to improve on it next year.
Still, although we're biased, we think it was a successful, well-executed project and a novel use of technology for news. Tom Giratikanon even gave the project his highest praise: both "worth doing" and "not the worst."
Here's a list of notable NYT 4th Down Bot links for the offseason (and if there are others, I'm happy to add them):
An intro post from Brian Burke when it was launched. "Even I'm tired of analyzing 4th downs."
A thoughtful critique from a statistician's perspective.
I talked about it on the Advanced NFL Stats podcast.
Neiman Lab wrote a feature on it.
Gigaom wrote a long post about it and then did a longer feature on Brian Burke, who developed the statistical model that drives the bot.
For some reason, it had something to do with brands and content strategy, though to be honest, this article makes little sense.
Nate Silver devoted a few paragraphs of his ESPN magazine cover story on the bot.
We're hoping to introduce a cousin or two this summer, too. If not, well, preseason is only 6 months away.