The Problem With Data Journalism

futurelab default header

Journalism has been thoroughly disrupted over the past decade. News organizations, especially newspapers, have come under heavy financial pressure, news bureaus have been closed or consolidated and journalists have had to rethink their profession.

Yet amidst the rubble a new form of the craft has emerged—data journalism. Rather than pounding a physical beat and cultivating human sources, this new breed immerses itself in statistical data and policy papers.

Two paragons of the new form, Nate Silver and Ezra Klein, have recently left traditional outlets to create their own news organizations, FiveThirtyEight and Vox, respectively. Yet, now it looks like the model might not be as scalable as it first appeared and these new ventures already seem to be struggling. Data journalism, for all its promise, has a problem.

The Making Of A Phenomenon

Neither Silver nor Klein came from a traditional journalism background. Silver began his career as an economic consultant, then took an interest in sabermetrics before he started handicapping elections. Klein got his start, ironically, by getting rejected by his college newspaper. Left to his own devices, he read up on policy papers.

Both created enormously successful personal blogs and were noticed by national media. Silver got a standing column at The New York Times and went on to account for as much as 20% of the traffic for the entire site. Klein created the Wonkblog at The Washington Post and reportedly generated 4 million page views per month.

Yet Silver and Klein, for all their success, are themselves a poor data set. Silver rose to prominence by predicting elections, a highly data rich environment. Klein has focused on policy, which lends itself to an academic approach. Most topics are not so easily analyzed. They often involve subtle context that is hard to discern from staring at a computer screen.

Now that both Smith and Klein have expanded their approach to a wider variety of subjects—and staffs far less talented than themselves—the seams in their approach have begun to show.

Data Divorced From Context

As Paul Krugman has pointed out with respect to Silver’s FiveThirtyEight, data only tells part of the story. Understanding data requires real world expertise. While Klein’s Vox employs an approach that is less quantitative and more qualitative, I’ve noticed similar problems, in particular with respect to its coverage of the crisis in Ukraine.

Vox’s foreign policy analyst, Max Fisher, has watched the crisis closely and reported on it diligently. In many ways his coverage has been everything you would want to see. It accurately reports relevant data and makes common-sense conclusions. Unfortunately, he has gotten almost everything wrong, as he himself has partly acknowledged.

The problem is that while he’s working with accurate data, his lack of expertise in the region causes him to misinterpret it. For example, working from language data and election results, he reports that Ukraine is divided more or less evenly between those see Ukraine as linked to Europe and those who see it as aligned with Russia.

Unfortunately the data, divorced from local context, leads Mr. Fisher to a specious conclusion and does him and his readers a great disservice. As Timothy Snyder, a scholar who has written a number of highly regarded books on the region, recently wrote:

Ukraine is a bilingual country. Electoral posters are in both languages. Candidates switch from one language to another on political talk shows. The giant banners on government buildings that read “One Country” are in both languages. If you watch a soccer game on television you might notice that the man doing the play by play speaks Ukrainian while the man doing color speaks Russian: almost all Ukrainians understand both and most speak both. If you go to a coffee shop you might find a polite waitress who adjusts to the language she thinks you speak best. No country in Europe is more cosmopolitan than Ukraine in this respect.

Now Snyder is a true expert on the region, but you wouldn’t have to be to know that Ukraine is a bilingual country—a reasonably observant tourist would pick that up. A little more research would reveal that the “pro-Russian” President Yanukovych had supported EU integration and only 41% of Crimeans (even less in Eastern Ukraine) wanted to join Russia.

Data separated from context is a dangerous thing—one of the few “Russian specialists” Fisher cites is himself a demographics analyst with little understanding of issues on the ground—and his coverage is riddled with misunderstandings that no one familiar with the region would make, such as a false controversy about what to call the country in question.

And Fisher is not a lone example. Nate Silver has also had to issue an apology because of misleading coverage. I only single Fisher out because he happens to report on a subject that I know well. In truth, it’s the data journalism model itself that falls short when confronted with complexity and nuance.

The Nature of Disruption

Data journalism is, in many ways, a disruptive innovation. The concept, developed by Harvard’s Clayton Christensen, describes a market where incumbents over-serve their consumers, opening up opportunities for new products that compete on the basis of a different value proposition.

Journalism can be an expensive proposition. You have to pay reporters to travel to where the story takes place, observe events in person, cultivate and interview countless sources. In comparison, performing statistical analysis of data sets or poring over policy documents is a relatively cheap and labor-saving way to operate.

At the same time, sometimes there are only so many facts on the ground and over-reporting can be tiresome—as when CNN engages in what Jon Stewart has described as “disaster porn.” So in that sense, data journalism offers a useful alternative perspective.

Yet traditional reporting still offers advantages that data journalism sorely lacks. In the case of Ukraine more traditional journalists like Slate’s Anne Applebaum, New Republic’s Julia Ioffe and Vice News’ Simon Ostrovsky—just to name a few—offer experience, expertise and on-the-spot reporting that data journalists couldn’t hope to match.

What The Future Holds For Data Journalism

The dirty little secret of disruptive innovation is that the disruptors themselves rarely prosper. Often, as in the case of MP3 players and digital cameras, their technologies are co-opted by larger players and then integrated into a more complete whole. Most probably, something similar will happen with data journalism.

There is nothing mutually exclusive about traditional reporting and data journalism. In fact, they are clearly complementary. Traditional journalists could benefit from more data literacy and objective analysis, while data journalism would be much improved by real world context.

Unfortunately, at this early stage, little integration has occurred. Traditional journalists often arrogantly dismiss the new breed as interlopers, while the data journalists seem too enthralled with their shiny new tools to do the hard work of cultivating sources.

Yet despite its shortcomings, data journalism still offers great promise. Both Silver and Klein are smart guys with incredible talent and many traditional news outlets have seen the advantages of the new approach. Hopefully, these problems will soon be resolved and journalism as a whole will be much improved.

Image via flickr

Original Post: http://www.digitaltonto.com/2014/the-problem-with-data-journalism/