The Recipe for Decoded

The goal of the NSA Decoded project was to create an interactive story that used all of the internet’s storytelling devices (video, interactives, maps, charts, text, and yes, GIFs!) to explain the story of the NSA files in an accessible and relatable way. The project was unique in that we were trying to wrap-up lots of our previous reporting, add new reporting, and deliver it in a way that takes a very complicated story, both technologically and legally, and makes it not only easy to understand, but yes, in fact, fun.

We also wanted to answer two questions we continually ran into while reporting these stories: “Why does it matter if I’m obeying the law?” and “What does this mean to me?”

One distinct challenge was the lack of visuals inherent to the story. We didn’t have any lovely video or photos to run with it, so we knew the design would be even more important. As such we started out with a fundamentally text-first approach seen across many projects over the past few years, with annotations (video/photo/document/etc.) that would appear alongside the text as you got to a particular point in the story.

Collecting the Ingredients

Over the past year or so, we’ve also developed a way of designing interactive projects that includes video storyboards. Greg Chen, our illustrator and animator, creates videos that take the flat elements we’ve designed and animate the elements to show the expected behavior:

Greg and I worked some on those for a week or so, while Feilding was out of town, then handed off some comps and a video to Feilding, who started building out the framework.

In the first month or so, that was what we had mocked up, that’s the framework we had built, and that’s the demo we were showing to the editors and reporters (full of lorem ipsum and selfies on green screens) so they could see what we were looking to do.

The Outline

With the design and framework going, we started outlining the content. (Yep. Just a written outline. Intense, right?) The outline encompassed all of the NSA revelations to date and organized by the bigger threads that have run throughout the reporting: collection, encryption, the FISA court, the results of the disclosures. Each of these topics had many bullet points below them. Then we took a look at each part of the outline, and decided where interactives/video/text/GIFs/whatever would work best to tell that bit of the story. For parts it was obvious: explaining TOR visually is much easier than writing about it. We ended up with something like 15 interactive elements or so and a handful of documents (some of course were new). The whole outline went through at least a dozen revisions.

Getting the Video

It took about a month to get the first few video interviews done. I could go on and on about how hard those were, but anybody who has shot video interviews knows the stress that goes into it: You have one shot. The video must be perfect. The audio must be perfect. The lighting needs to be perfect. The questions need to be provocative. The hope is that the interviewee is engaging and well-spoken. You have to find a good, quiet space (ideally not your apartment). Oh yeah, and then do it on a green-screen. It was tough. Thank goodness for Bob Sacha, videographer and friend, who helped us out.

It’s Great—Now Change Everything

After a month we had a few of the interviews done and realized just how well the direct-to-camera quotes were working, and we had some of the graphics pretty far along, and they were looking great—so we decided to rethink the entire format of the project. We pulled the talking heads out of the sidebars and inserted them directly into the narrative. Same with the interactives. And with the encouragement of Janine Gibson (editor in chief), Stuart Millar (deputy editor), and Ewen MacAskill (reporter), we decided to write into and out of the videos and graphics, essentially complementing the multimedia with copy. This was an truly liberating and infinitely flexible approach, as we were no longer beholden to any type of media to tell the story, which I think was a big part of why the project turned out to feel so cohesive.

Also at this point, we realized we had so much content that if we were to add 10,000 words to it, it would just be too much! We were aware of previous efforts at long-form multimedia and the responses they’ve received, common complaints included that some were far too long while others contained “bells and whistles” that distracted from the story, and we also just felt since our goal was to engage and inform, we wanted to keep the reader moving throughout the content. So our decision to highlight the multimedia and add text when appropriate worked out nicely.

Putting It All Together

The final bit was selecting quotes from the interviews for each individual section of the outline. We transcribed all of the interviews as we went along (this is vital) and then Ewen and I went through all of them selecting the best quotes for each section.

In the end, the project spanned about three months. A lot of that time was scheduling and shooting interviews, which required a lot of back and forth to DC, and probably delayed the launch by a few weeks. Greg, Feilding, and I worked on it pretty much straight for two months, and Kenan Davis and Kenton Powell, both of whom joined the team after the project already started, jumped in for a month or so each. When we did the interviews we worked with various reporters throughout the newsroom, especially Ewen MacAskill, Spencer Ackerman, and Paul Lewis.

Is There a Template?

There is no template for this project. But while we’re on the topic, I would love for somebody to share the template for the next Pulitzer prize-winning investigative piece that is being written right now, because we’d love to use that in our newsroom. (Ahem.)

The truth is, this kind of journalism is bespoke, like most of the best journalism. Each story requires its own consideration. And while I’d like to think we pushed an innovative way to tell these stories, it’s not like we can just plug the next one in and be done in a few days. As Feilding will attest in the following explanation of the project’s technicalities, we invented no new wheels. The technology is what it is, and we’ve certainly built a reusable framework, but where the value lies is in the content, and there’s no template for content.

But you all really came to Source to see how the cake is made, amirite?

Baking the Cake

The evolving nature and size of this project created several technical requirements for both production and consumption. First, the project had to be responsive to fit and load content appropriate for the device. Second, multiple people needed to be able to build interactives at the same time without dealing with excessive merge conflicts. Lastly, elements would need to be easily rearranged during the editing process.

A Flexible Framework & Workflow

Decoded is built on a lightweight framework that treated each interactive element as a separate project consisting of its own JavaScript module, less (compiled to CSS), and HTML. The framework itself consisted of some HTML scaffolding, narrative text, and Ruby ERB template references pointing to each of the interactive elements. In addition to the framework, each module, could be developed, browser tested, and edited independently. A rendering script built in Ruby stitched together the framework and modules, creating a list of JavaScript assets managed by a global progressive loader, minified CSS, and a single HTML page containing all of the assets.

A final Ruby packaging script applied gzip compression to CSS and JavaScript files and necessary file headers during the upload process to S3. The packager also created a final HTML asset with all paths configured to point to the production CDN.

The render and packaging scripts enabled a workflow that was fundamental to the development and editing processes: Elements could be viewed on mobile, tablet and on Internet Explorer to make style adjustments and debug JavaScript. The output of the render scripts allowed each piece to be viewed and edited in context. Testing for Decoded occurred throughout the development process as elements neared completion. Furthermore, the workflow allowed developers to adjust elements for mobile rendering and progressive enhancements in desktop browsers.

Ultimately, it was the workflow and the ability to treat elements as independent modules that was essential. Decoded uses open source technology one might expect: D3, jQuery, Video.js, and LESS.

Dealing with the Video

The major technical challenge presented by Decoded was dealing with more than ten HTML5 video players on a single page. To solve this, the progressive loader, which loads videos and other assets before they’re visible to the user, also progressively unloads videos after they scroll out of the viewport. Furthermore, the progressive loader also turned off rendering for processor-intensive elements like the cables graphic once they left the viewport.

Ongoing Improvements

For Decoded, the takeaway we’re already working on is creating a template to publish on theguardian.com that does not include the standard site wrapper and page furniture. Currently, the Decoded page loads as a standard interactive template with corresponding site header and footers, styles, and JavaScript. For Decoded, our global JavaScript, through a function called destroyer, gracefully removes these unused fragments of the current site html, along with the global site CSS. While doing this doesn’t cause any dramatic problems, creating a stripped-down template will allow for improved optimization on future projects by not loading JavaScript and HTML that is never used.

We hope you enjoyed the cake!