Empire: Lessons from Pushing the Boundaries of Web Video
We at POV recently released a series of interactive documentaries that use technologies like HTML5 video to tell nonfiction stories in new ways. One of these documentaries, Empire, explores the long-term, unintended consequences of Dutch colonialism. The piece, directed by Kel O’Neill and Eline Jongsma and coded primarily by Genevieve Hoffman. The piece began as a small prototype, hacked together under the time pressure of a POV Hackathon with other technologists, and the team built it into an ambitious and fully functional interactive experience. I had the opportunity to help out in the final stages with some of the technical challenges of deploying a media-heavy piece like this on the web.
In interactive pieces like Empire, the viewer’s browser is essentially performing the work of a post-production suite, rendering a movie in real time over an unreliable, remote network connection. So small quirks in browser implementations or device limitations can become major stumbling blocks in the development and debugging process.
Pre-Loading Video, But Which One?
The first big challenge was getting all of the media components to load reliably and quickly. Empire includes eight video files, two audio files and many images, totalling hundreds of megabytes. With this much media to load, it became necessary to balance the need for a fast start-up time with the need to minimize buffering once a video has started to play. Empire consists of multiple chapters that a user can switch between at any time, so it is impossible to predict which media elements will be played next. It sounds tempting to pre-load everything, but browsers aren’t able to load many large files simultaneously. And web developers have limited control of how and when the browser loads video data from the network. For the most part, how much data to load is decided by the browser, which guesses how much you’re going to need, and the algorithm is different for each browser.
We found the best solution was to pre-load just the “metadata” of all the media files (the header of a media file that contains the duration, the pixel dimensions and enough information for the browser to know if it can play it) and then start loading the full file once the user selected a chapter. With only a few files to load at a time, we see that the start-up time is not more than a few seconds and there is minimal pausing to buffer, even over a mobile 3G connection.
The video and audio tags have an attribute called “preload” that we set to metadata so the browser knows not to start loading anything beyond that until we tell it to. Our video tag looks like this:
<video id="video1" preload="metadata">
<source src="videos/video1.mp4" type="video/mp4"/>
<source src="videos/video1.webm" type="video/webm"/>
var video1 = document.getElementById('video1');
From this point, the loading process is beyond our control as the browser progressively loads as much of the video as it thinks we need based on whether the video is actually playing and how fast the data comes over the network.
The Empire Framework
Keeping track of the state of things can get quite complicated. Certain components may be visible at different times, each of the media elements load in different ways, and there are many elements that the user can select at any time. There is no guarantee that these events will ever happen in the same order twice, and we needed to be sure that images, video and audio were only visible and audible when they’re supposed to be.
In my next post, I’ll share code for a key part of Empire’s “Migrants” chapter, in which we enabled synchronized video playback for shared viewing experiences.
[Cross-posted to the POV Tech Blog.]