Features:
Introducing Elex, a Tool to Make Election Coverage Better for Everyone
This new collaboration from the New York Times and NPR needs your help
It could be a new day, for your election data. (Kheel Center)
“End the elections arms race” has become a rallying cry in American data journalism. Many newsrooms spend tremendous resources writing code to simply load and parse election data. It’s time we stopped worrying about the plumbing and started competing on the interesting parts. We decided it was time we put some code against our beliefs – our contribution is a tool we’re calling Elex.
What Is It?
Elex is a command-line tool to get election results from the Associated Press Election API v2.0. Elex is designed to be friendly, fast and agnostic to your language/database choices.
We were inspired by the Los Angeles Times Data Desk and their Python Elections library. Our goal was to take the good ideas from Python Elections and develop something compatible with the new version of the API provided by the AP and make a tool that can easily integrate into non-Python environments.
Why Did You Write Elex from Scratch?
Late in the 2015 election cycle, the Associated Press debuted a new version of its election results API. It returns data over HTTP, in a very different way from the FTP-based system that we’ve used for years. The software you’ve been using probably won’t work for the 2016 cycle without an overhaul.
We saw this as an opportunity to rethink how we retrieve and store election results. For example, at the New York Times, our code that loaded election results was tightly coupled with the administrative UI (for editing names, calling races, etc.), and the user-facing UI.
However, we don’t take this task lightly. Our old code was battle-tested. Starting over is always a risk. So, as part of our development effort, we took painstaking care to test and use Elex in real situations.
How Do I Get Started Using Elex?
Install
You’ll need a valid API key from the Associated Press and a standard Python installation (if you’re on OS X, follow chapters 0 through 3 of NPR’s installation guide).
Fire up your command-line app, and type:
$ mkvirtualenv elex
$ pip install elex
$ export AP_API_KEY=<YOURAPIKEY>
Get Data
Elex is a command-line tool, which means there’s very little configuration. You only need to type a single command to begin loading results. Here, we ask Elex to return results from the November 2015 general election as CSV and grab the first three rows.
$ elex results 2015-11-03 | head -n 3
2015-12-01 21:18:06,454 (INFO) elex (v0.0.42) : Getting results for election 2015-11-03
id,unique_id,raceid,racetype,racetypeid,ballotorder,candidateid,description,fipscode,first,incumbent,initialization_data,is_ballot_position,last,lastupdated,level,national,officeid,officename,party,polid,polnum,precinctsreporting,precinctsreportingpct,precinctstotal,reportingunitid,reportingunitname,runoff,seatname,seatnum,statename,statepostal,test,uncontested,votecount,votepct,winner
7584-polnum-9037-1,polnum-9037,7584,Special General,G,1,12484,Jeffco Successor Direct,,Brad,False,False,,Rupert,2015-11-04T03:00:44Z,state,False,0,School Board,NP,,9037,0,0.0,0,1,,False,District 1,1,Colorado,CO,False,True,0,0.0,False
7584-polnum-9037-6030,polnum-9037,7584,Special General,G,1,12484,Jeffco Successor Direct,8059,Brad,False,False,,Rupert,2015-11-04T03:00:44Z,subunit,False,0,School Board,NP,,9037,0,0.0,0,6030,Jefferson,False,District 1,1,Colorado,CO,False,True,0,0.0,False
$ elex results 2012-11-06 > results-2012-11-06.csv
2015-12-01 21:21:45,749 (INFO) elex (v0.0.42) : Getting results for election 2012-11-06
Why Should I Use Elex?
It does one thing well
We decided to adhere to the UNIX philosophy—that each program should do one thing well. Elex is a self-contained command-line tool. To use it, you only need an internet connection and an AP elections API key. It returns only JSON or CSV. Elex is designed to be a utility, not an all-in-one elections application.
It is (mostly) agnostic about data storage and your software stack
Elex is written in Python. But because it is a command line tool that outputs CSV and JSON, it works with any database that can consume CSV or JSON and integrates with any programming language that can call a shell command.
It’s Fully Documented
Elex has a full suite of documentation both on the web and within the code itself. We focused not just on explaining what’s happening under the hood but also on practical matters like recipes and usage tips.
It Has a Test Suite
Elex features a full suite of tests that give you confidence that the code is doing what it ought to do. Anywhere we make a calculation, there’s a test. Anywhere we create data, there’s a test.
It’s developed in public and has an open-source license
Elex was developed entirely in public. Don’t like the way we do it? Fork our code and send a pull request. Want to do something else altogether? Take our code and re-use it with the confidence that it’s licensed under the Apache License, version 2.0.
It’s Pretty Fast
Our old systems took a minute or more to parse, load, and deliver county-level data to the client. Elex cuts that time to less than 20 seconds using Python 3 and less than 30 seconds using Python 2.7. And we’re working on new ways to further speed it up.
We Need You!
We want you to rock out with us. Elex is stable but immature. To get the tool ready for the big elections coming in 2016, we need you to help us polish and test it.
Give Us Your Recipes
Elex is designed to be part of a pipeline that delivers election results to your audience. We want developers implementing Elex to share their recipes for loading and displaying data, and to help improve our reference implementation.
A few ideas for recipes we’d love to see tested and shared:
Loading data into MySQL, SQLServer, and other popular traditional relational databases
Loading data into ElasticSearch, MongoDB, and other specialized databases
Handling name overrides and race calls
Integrating with Python frameworks like Django and Flask
Integrating with Ruby frameworks like Rails
Integrating with Node frameworks
Processing data with jq and csvkit
Writing directly to S3 or other cloud storage providers
So let’s get this busywork out of the way so we can tackle the really hard problem: Delivering valuable news!
Credits
-
Jeremy Bowers
Jeremy Bowers is a developer on the Interactive News Team at The New York Times and previously worked for NPR, the Washington Post, and the St. Petersburg Times. He spends his weekends watching baseball and obsessively honing his ramen broth. As a college debater, he spoke more than 600 words per minute. (Photo credit: Claire O’Neil, NPR Visuals.)
-
David Eads
David Eads is a news applications developer at ProPublica Illinois, where he combines journalism with software development. Ever since he built the website for his high school newspaper in 1996, Eads knew he wanted to work at the intersection of media and technology. He moved to Chicago for college in 1999, studying physics at North Park University. During school he helped found the Invisible Institute, where he also maintained a blog about Chicago public housing called The View From The Ground. He later helped create FreeGeek Chicago, a community-based computer recycling organization and the Supreme Chi-Town Coding Crew, a weekly workshop to teach data journalism. He’s also worked on visual journalism teams at the Chicago Tribune and, most recently, at NPR Visuals.
