All About the dailygraphics Rig from NPR

A Q&A with Alyson Hurt and Christopher Groskopf

Last week, NPR’s Visuals team released their dailygraphics rig, which offers workflow for small-scale visualizations, interactives, and graphics, along with “automated machinery for creating, deploying and embedding these mini-projects.” Their introductory blog post breaks down how to set up and use the rig, and the code is open source and ready to use. Alyson Hurt joined last week’s OpenNews community call to talk a little about the project, and we chatted with her and Chris Groskopf about how the rig came to be, what kind of skills are required to use it, and their aim to improve code quality and culture through process-improving tools.

The Problem It Solves

Q. Before we get into the code, can you say a little about why you developed the rig? What are the benefits to adopting this for simple visualizations?

Alyson Hurt: This rig is intended not just for visualizations, but for any small project that needs to be embedded in a larger site. We’ve used it for graphs, certainly, but also data tables, before-and-after sliders and small interactives. We’re even building it into a workflow for managing Instagram photo call-outs!

So, before the News Apps/Visuals team formed at NPR, we were a little Wild West about deploying editorial projects. We edited files directly on the server (mounted like a fileshare) and there was no version control. With News Apps, the app-template and the workflow that developed around that, we moved into a more structured process with version control and command-line deployment for larger projects that live outside of the CMS. And then this year, we started applying some of that process to how smaller, short-turnaround projects (we call them “daily graphics”) are produced.

Each new mini-project starts with a baseline set of files and code, and then I work from there depending on that particular project’s needs. I do reuse a lot of code from past projects. Maybe at some point, we’ll add a few more starter templates—instead of just “fab add_graphic:foo” we’d have, say, “fab add_column_chart:foo” and “fab add_photo_callout:foo”. We’ll see what makes sense over time.

Under the Hood

Can you say more about the mini-CMS features that the rig enables for simple interactives and visualizations?

AH: Each mini-project can be linked to a Google Spreadsheet. We’ve used Google Spreadsheets as a kind of low-fi CMS for our larger projects, and it’s worked pretty well for us. It allows us to separate text from code, so an editor can enter or copyedit content while I set up the rest of the project — and there’s no worry of a DIV tag accidentally getting nicked or anything like that.

It’s also great for populating basic data tables. There are only so many TR/TD tags I can stand to write! (There’s an example of the template markup at the bottom of this post about responsive tables on our team blog.)

And you can use multiple sheets in your Google Spreadsheet, each serving a different purpose. One sheet might be a data table that you loop over, and another sheet might be more one-off chunks of text, like a headline or credit.

Did any interesting challenges arise on the dev side as you built out the rig?

Chris Groskopf: This project was a fairly natural evolution of our existing app-template, so we had already solved most of the problems. Probably the two most interesting pieces from a technical perspective are the responsive iframes javascript library (that we have since factored out as pym.js) and the little bit of Python magic that allows each graphic to have its own configuration. Without going into too much detail on the latter, our rendering infrastructure will dynamically import a file called graphic_config.py if it exists and then use the contents of that file to configure the graphic template. In particular, we use this to specify a unique Google spreadsheet for each graphic that uses the copytext part of the rig.

AH: I think another challenge, just workflow-wise, was to keep this as lightweight as possible for the user. For one, most potential users of this rig on our team are beginning-to-intermediate-level coders (including those of us who make graphics and a couple photo editors). Certainly, it takes time to get comfortable with the command line, but this can’t be horribly complicated. Second, a lot of the work that we do with dailygraphics is on-deadline stuff, and we need to be able to publish (and republish last-minute edits) with minimal overhead. I think we’ve achieved that.

Also, since this rig is used almost daily, we find all kinds of interesting edge cases (or just plain bugs) that inform changes to the overall rig, or to components like copytext. So we’re constantly improving on this, which is great.

As a sidenote, the two-repo setup that keeps the “machinery” open while protecting NPR’s copyrighted or embargoed content is kind of ingenious. How did you end up with this solution?

CG: This is something we’ve struggled with frequently at NPR. We want to open source everything, but for perfectly legitimate reasons our lawyers don’t want us implying that the source code license applies to the content (text, images, etc.) by having them in the same repository. Previously, this caused us to develop both copytext and the assets rig, both of which are now part of dailygraphics.

For dailygraphics the problem was compounded because the source code effectively is the content in pretty much every case. Moreover, we’re building charts and tables, which are also frequently things we don’t want to slip out before we’re ready. To that end we looked at a variety of ways of segmenting the repository. Initially I had thought we would use a git submodule, but ultimately it ended up being far simpler for other users of the rig if we just set up two repositories and handled the connection between them with configuration. This also has the ancillary benefit of making it very easy for new users to get started.

Getting Set Up

If I’m in a newsroom that hasn’t yet adopted something like the App Template, what kind of set-up am I getting into if I want to start using this? What kind of skills do I need?

CG: In general, if you have the skill to build a web graphic with HTML, CSS and JavaScript, then you probably have the skills you need to use this rig. Beyond web programming fundamentals, you’re going to need an Amazon Web Services account so that you can use S3, which is where the rig deploys the graphics. You’ll also need enough command line familiarity to setup your S3 credentials, checkout the project from Github and install the requirements. If you want to use our copytext component for managing text, you’ll need to have a Google Drive account.

That’s really it. We go out of our way to work with easily accessible technologies, so if you’re doing any web development then there is a very good chance you already know everything you need to know in order to use it.

Is there anything especially tricky to keep an eye on here as a potential user? Common blunders?

AH: I think we’ve ironed out any big trip-ups over the past few months of using it. The main one for me since we’ve split up the repo between dailygraphics machine and actual graphics is managing my terminal tabs — committing to GitHub from one tab, but doing everything else from another tab. But that’s fairly minor.

Change the Code, Change the Culture

One of the things that’s most interesting about a lot of the code coming out of NPR these days is that it’s sort of a carrier wave for culture and good practices—in this case, for treating even small projects in a clean, standard way. How much of that is conscious? And what kind of impact are you seeing from this approach?

CG: It’s very, very conscious. I take the craft of software development very seriously and I believe that there is an intrinsic danger in treating small things as unworthy of being done well. If all of our projects are important then they all deserve to be done thoughtfully. If they aren’t important then we shouldn’t do them. I spoke about my ardent desire for code quality in my NICAR 2014 lightning talk this year. As I have said to several people who have asked me about that talk, I saw that as a first salvo in a mission to insist on quality software development practices in our community. This year especially, I’ve tried to ensure we are good role models for how to code cleanly, write good documentation, and provide transparency into our methods. Fortunately, everyone on the Visuals team shares an amazing commitment to quality and I think we’ve all seen the tangible benefits of working this way. All that documentation we write is for us too. And all those blog posts we write clarify our thinking and provide fodder for the next iterations. Everything we do is about reducing our own internal dependencies, including turning our code out into the world for everyone else to poke holes in.

Thanks, Alyson and Chris!





  • Christopher Groskopf

    Christopher Groskopf is a data journalist on the NPR Visuals team. In November he’ll be joining Quartz. He divides his time between Tyler, TX and Washington, DC. Find him on Twitter: @onyxfish.

  • Alyson Hurt

    Alyson Hurt is the graphics editor at NPR, serving on the Visuals team (formerly News Apps + Multimedia). Previously at the Washington Post and the Arizona Republic. Graduate of ASU’s Cronkite School and Georgetown’s CCT program.

  • Erin Kissane

    Editor, Source


Current page