“End the elections arms race” has become a rallying cry in American data journalism. Many newsrooms spend tremendous resources writing code to simply load and parse election data. It’s time we stopped worrying about the plumbing and started competing on the interesting parts. We decided it was time we put some code against our beliefs – our contribution is a tool we’re calling Elex.

What Is It?

Elex is a command-line tool to get election results from the Associated Press Election API v2.0. Elex is designed to be friendly, fast and agnostic to your language/database choices.

We were inspired by the Los Angeles Times Data Desk and their Python Elections library. Our goal was to take the good ideas from Python Elections and develop something compatible with the new version of the API provided by the AP and make a tool that can easily integrate into non-Python environments.

Why Did You Write Elex from Scratch?

Late in the 2015 election cycle, the Associated Press debuted a new version of its election results API. It returns data over HTTP, in a very different way from the FTP-based system that we’ve used for years. The software you’ve been using probably won’t work for the 2016 cycle without an overhaul.

We saw this as an opportunity to rethink how we retrieve and store election results. For example, at the New York Times, our code that loaded election results was tightly coupled with the administrative UI (for editing names, calling races, etc.), and the user-facing UI.

However, we don’t take this task lightly. Our old code was battle-tested. Starting over is always a risk. So, as part of our development effort, we took painstaking care to test and use Elex in real situations.

How Do I Get Started Using Elex?

Install

You’ll need a valid API key from the Associated Press and a standard Python installation (if you’re on OS X, follow chapters 0 through 3 of NPR’s installation guide).

Fire up your command-line app, and type:

$ mkvirtualenv elex
$ pip install elex

$ export AP_API_KEY=<YOURAPIKEY>

Get Data

Elex is a command-line tool, which means there’s very little configuration. You only need to type a single command to begin loading results. Here, we ask Elex to return results from the November 2015 general election as CSV and grab the first three rows.

$ elex results 2015-11-03 | head -n 3

2015-12-01 21:18:06,454 (INFO) elex (v0.0.42) : Getting results for election 2015-11-03

id,unique_id,raceid,racetype,racetypeid,ballotorder,candidateid,description,fipscode,first,incumbent,initialization_data,is_ballot_position,last,lastupdated,level,national,officeid,officename,party,polid,polnum,precinctsreporting,precinctsreportingpct,precinctstotal,reportingunitid,reportingunitname,runoff,seatname,seatnum,statename,statepostal,test,uncontested,votecount,votepct,winner

7584-polnum-9037-1,polnum-9037,7584,Special General,G,1,12484,Jeffco Successor Direct,,Brad,False,False,,Rupert,2015-11-04T03:00:44Z,state,False,0,School Board,NP,,9037,0,0.0,0,1,,False,District 1,1,Colorado,CO,False,True,0,0.0,False

7584-polnum-9037-6030,polnum-9037,7584,Special General,G,1,12484,Jeffco Successor Direct,8059,Brad,False,False,,Rupert,2015-11-04T03:00:44Z,subunit,False,0,School Board,NP,,9037,0,0.0,0,6030,Jefferson,False,District 1,1,Colorado,CO,False,True,0,0.0,False

$ elex results 2012-11-06 > results-2012-11-06.csv

2015-12-01 21:21:45,749 (INFO) elex (v0.0.42) : Getting results for election 2012-11-06

Why Should I Use Elex?

It does one thing well

We decided to adhere to the UNIX philosophy—that each program should do one thing well. Elex is a self-contained command-line tool. To use it, you only need an internet connection and an AP elections API key. It returns only JSON or CSV. Elex is designed to be a utility, not an all-in-one elections application.

It is (mostly) agnostic about data storage and your software stack

Elex is written in Python. But because it is a command line tool that outputs CSV and JSON, it works with any database that can consume CSV or JSON and integrates with any programming language that can call a shell command.

It’s Fully Documented

Elex has a full suite of documentation both on the web and within the code itself. We focused not just on explaining what’s happening under the hood but also on practical matters like recipes and usage tips.

It Has a Test Suite

Elex features a full suite of tests that give you confidence that the code is doing what it ought to do. Anywhere we make a calculation, there’s a test. Anywhere we create data, there’s a test.

It’s developed in public and has an open-source license

Elex was developed entirely in public. Don’t like the way we do it? Fork our code and send a pull request. Want to do something else altogether? Take our code and re-use it with the confidence that it’s licensed under the Apache License, version 2.0.

It’s Pretty Fast

Our old systems took a minute or more to parse, load, and deliver county-level data to the client. Elex cuts that time to less than 20 seconds using Python 3 and less than 30 seconds using Python 2.7. And we’re working on new ways to further speed it up.

We Need You!

We want you to rock out with us. Elex is stable but immature. To get the tool ready for the big elections coming in 2016, we need you to help us polish and test it.

Give Us Your Recipes

Elex is designed to be part of a pipeline that delivers election results to your audience. We want developers implementing Elex to share their recipes for loading and displaying data, and to help improve our reference implementation.

A few ideas for recipes we’d love to see tested and shared:

• Loading data into MySQL, SQLServer, and other popular traditional relational databases

• Loading data into ElasticSearch, MongoDB, and other specialized databases

• Handling name overrides and race calls

• Integrating with Python frameworks like Django and Flask

• Integrating with Ruby frameworks like Rails

• Integrating with Node frameworks

• Processing data with jq and csvkit

• Writing directly to S3 or other cloud storage providers

So let’s get this busywork out of the way so we can tackle the really hard problem: Delivering valuable news!

• Share on Twitter
• Share on Facebook