Articles

Projects walkthroughs, tool teardowns, interviews, and more.

Articles tagged: data

How We Made the SOTU Twitter Visualization

By Nicolas Belmonte and Simon Rogers

Posted on March 18, 2014
People tweet what they think, when they think it—and, crucially, we wanted to provide a visualization for the State of the Union speech which reflected that. This wouldn’t be a (shudder) word cloud based on frequencies but a way to track the conversation on Twitter as it was directly influenced by the President’s speech.
You Got the Documents. Now What?

By Jonathan Stray

Posted on March 13, 2014
Jonathan Stray’s guide to turning documents into data you can run with.
Human-Assisted Reporting Gets the Story

By Tyler Dukes

Posted on February 27, 2014
Tyler Dukes on combining the power of data-sorting tools with old-fashioned digging.
How to Make a News App in Two Days

By Al Shaw

Posted on January 31, 2014
As part of the orientation week for the 2014 class of Knight-Mozilla OpenNews Fellows, fellow nerd-cuber Mike Tigas and I led a hackathon at Mozilla’s headquarters in San Francisco…
To Scrape, Perchance to Tweet

By Abe Epton

Posted on January 14, 2014
At the Chicago Tribune, we had a simple goal: to automatically tweet contributions to Illinois politicians of $1,000 or more, which campaigns are required to report within five business days. To see, in something approximating real time, which campaigns are bringing in the big bucks and who those big-buck-bearers are. The Illinois State Board of Elections (ISBE) has helpfully published exactly this data for years online, in a format that appears to have changed very little since at least the mid-2000s. There’s no API for this data, but the stability of the format is encouraging. A scraper is hardly an ideal tool for anything intended to last for a while and produce public-facing data, but if we can count on the format of the page not to change much over at least the next several months, it’s probably worth it.
Introducing Treasury.IO

By Michael Keller and Cezary Podkul

Posted on January 7, 2014
The U.S. Treasury’s Daily Treasury Statement lists actual cash spending down to the million on everything the government spent money on each day, as well as how it funded the spending. But, the Treasury only releases these files in PDF or fixed-width text files like this one, making any analysis very difficult. To liberate the data and make it easy to analyze federal money flows across time, we created Treasury.IO. The system we built downloads and parses the fixed-width files into a standard schema, creating a SQLite database that can be directly queried via a URL endpoint.
Open Your Data

By Waldo Jaquith

Posted on November 21, 2013
Waldo Jaquith on the whys and wherefores of making it open
And Remember, this Is for Posterity

By Jacob Harris

Posted on November 14, 2013
Jacob Harris on the hows and whys of designing interactives to survive the future.
Building News Apps on a Shoestring

By Alan Palazzolo

Posted on October 31, 2013
Alan Palazzolo on how the MinnPost team rocks it without a big budget
Well Hello, Census

By Joe Germuska

Posted on October 17, 2013
Joe Germuska on the iterative, human-centered process that’s made the new Census Reporter project especially awesome.
Let research apps, MVC JavaScript, and APIs work for you

By Agustin Armendariz, Michael Corey, and Aaron Williams

Posted on October 14, 2013
The Center for Investigative Reporting continues their work visualizing Department of Veterans Affairs’ data. Here, they discuss their development process.
Using Big Data to Ask Big Questions

By Chase Davis

Posted on October 4, 2013
Chase Davis lays down some data science upon us to change how you think about the questions you’re asking of your data
US Shutdown Scuttles Data

By Dan Sinker

Posted on October 3, 2013
As the government shutdown grinds into its third day, many news developers, civic data hackers, and open gov activists are starting to feel the hurt due to the suspension of most government data feeds, APIs, and websites. How they’re adapting and collaborating to fill the gaps of the shutdown.
The Times Regrets the Programmer Error

By Jacob Harris

Posted on September 19, 2013
Jake Harris opens a serious barrel of monkeys about when and how to issue corrections for data journalism.
Kill All Your Darlings

By Matt Waite

Posted on September 12, 2013
Matt Waite on what to do when things don’t work out like you planned.
All About Reporter

By Erin Kissane and Jeremy Singer-Vine

Posted on September 10, 2013
The Wall Street Journal’s Jeremy Singer-Vine recently released Reporter, an open source tool that makes it easy to hide and reveal the code behind common forms of data visualization presented on the web. We spoke with him about the tool’s makeup, design goals, and future development plan.
People Power Prevails!

By John Keefe

Posted on September 5, 2013
John Keefe on tracking the cicada pestilence with open source sensor journalism and crowdsourced data collection
Telling Your Boss “No”

By Matt Waite

Posted on August 22, 2013
Matt Waite says just because you can make it doesn’t mean you should
Sane Data Updates Are Harder than You Think: Part 3

By Adrian Holovaty

Posted on August 15, 2013
Third in a three-part series by Adrian Holovaty about hairy data-parsing problems from a journalist’s perspective
They Are Tweet Zombies!! They Are Followers!!

By Jacob Harris

Posted on August 8, 2013
Jake Harris on how dead accounts and spambots can mess with your Twitter data mojo