Live Streaming History
How the Texas Tribune Handled the SB 5 Filibuster Traffic
Last night I boarded a plane in Boston to travel back home as Texas state Sen. Wendy Davis approached five hours into her 13 hour attempt to filibuster SB 5, abortion legislation that she opposed. The last analytics I saw from my phone showed several thousand users watching our stream, but I wasn’t worried about the capacity of our site to handle the load.
I arrived in Texas a few hours later as everything heated up. Our traffic had doubled while I was in the air and we had over 100,000 people watching our live stream of the Texas Senate proceedings and we weren’t done. Before the night was over we had over 15,000 concurrent users on texastribune.org and more than 183,000 people watching our YouTube live stream on various places around the net as the proceedings wound down shortly after midnight local time (we peaked at 12:03, as the final votes were cast too late).
We handled all of this traffic with no downtime and no additional servers.
Applying a Layer of Varnish
Shortly after Rick Perry entered the 2012 Presidential race, he changed a long-held stance on the HPV vaccine and our coverage of that story set the record for most concurrent users on our site. Huffington Post picked up the story and at one point we were their top headline on the site.
We scrambled all day to keep our servers up and running. All of our servers are on Amazon Web Services so we could scale out, but we had so many servers running we were crushing our database. We needed a way to serve our site and protect our app servers. Enter Varnish.
Varnish sits in front of our Armstrong/Django-powered app servers and caches the responses it gets from them. Its entire purpose is to retrieve some piece of content from somewhere on the Internet, cache it locally in memory, then serve it back as fast as possible and its amazing at that job. Once the period of caching has expired (five minutes by default in our configuration, but each of our views can set their own criteria), Varnish makes another request of the app server to regenerate the page.
The secret of Varnish is the concept of grace: instructions to serve expired content when it can’t regenerate it for some reason. The length of the grace period is configurable, but we use 48 hours. That means our site’s backend could be down for as long as two days and the only indication the public would have is that the content isn’t updating.
Varnish allowed us to serve our homepage and live stream straight out of memory. Every five minutes, Varnish regenerated the both of those pages while still serving the old version until the new version has been successfully regenerated (within seconds). We only did two requests every few minutes to regenerate each of those pages. Our app servers had no idea anything was different than any other Tuesday night.
That’s only half the battle, though. We had nearly 12 times that much traffic on our video live stream.
Stand on the Shoulders of Giants
This part isn’t nearly as interesting. We stayed up because we hosted it on YouTube. We’re not video experts. We grok the web, but streaming live video to hundreds of people, much less hundreds of thousands of them, is hard. Rather than recreate the wheel and try to provide our own stream with a custom player, we’ve offloaded all that work to YouTube.
Interestingly enough, this is similar to the model that SCOTUSblog has used the last few days as they’ve been slammed. Rather than try to come up with their own custom live blog, they’ve simply dropped in CoverItLive. Figure out where your strengths are and play to those. Ours is definitely not video, much less live, streaming video, so we outsource it.
To make all of this happen, we have two Mac Mini servers in a server rack at the Texas Capitol. Each has a Blackmagic Intensity Extreme that converts the raw video and audio feeds from House and Senate into a digital format. Once the conversion is done and fed to the Mac Minis, we ship it off to YouTube.
Our setup is simple and efficient. We offered the world a glimpse into Texas politics at work using consumer hardware and less than a few hundred kilobytes a second of uploaded data.
Protect your application servers. The bottom line is that you can’t handle massive volumes of scale and serve all of your content dynamically all of the time. You don’t have to strip your site down, either, just use what’s available to you. Use Varnish to protect your app servers and make sure that they don’t get slammed. Use external services for the really hard parts so you don’t have to worry about that.
Code & JIRA Gardener, Biased towards action. Subversively assertive. Food whisperer. Digital polyglot. Maker of trouble. Too opinionated for my own good.