A few years ago, a group of economists led by Nobel prize winning Joseph Stiglitz tried to develop measurements of societal well-being beyond the standard metric of GDP, which only measures a narrow spectrum of economic activity. Beyond any of their specific recommendations, though, their report brought to light the history of GDP, a history that is incredibly instructive for the coming age of big data. The story? GDP was never meant to serve as a proxy for social well-being, but simply as a way to track economic output.
Over time, and with few other tangible metrics that give any sense of how society is doing, GDP has inadvertently become the default measurement that governments use to evaluate how much, or little, they’re doing to enhance well-being. As Stiglitz wrote at the time, "what we measure affects what we do." And so that measurement, of economic output, has had an outsized impact on how governments have functioned, at the expense of policies that might focus, instead, on improving health, happiness, and environmental quality.
Now this might seem like it isn’t connected to the idea of big data, where the promise seems to be that in contrast to past limits, we will be able to track, measure, and understand just about anything we’d want to know about our behavior as individuals and as a society. With the amount of information in the world forecast to increase by a multiple of 44 between 2009 and 2020, most concerns center around how we’ll keep from becoming overwhelmed by information.
Here’s one likely way we’ll cut through the deluge: We’ll have so much information that seems important at any given point that we’ll gravitate toward clear measurements—and have increasingly less tolerance for the ambiguous. For most of us, if we try something, and it doesn’t quickly demonstrate that it’s working, we’ll move on.
Now, this may not sound that bad. But it’s a bias that has the potential to increasingly push toward favoring the short-term over the long-term, with potentially disastrous consequences.
One of the worst, recent examples of short-term changes in data causing major problems comes from food. Between 2006 and 2008, global rice prices doubled, plunging hundreds of millions of people worldwide into hunger. Among the causes? Some local weather disruptions started pushing prices up (PDF), and seeing prices creeping upward, governments and consumers started buying and hoarding large supplies, creating a vicious cycle that ultimately helped push the number of people battling hunger over the one billion mark.
What we measure affects what we do, and in that case, the ability to detect a short-term price increase and react to it made the effects of that short-term challenge much more severe. Here in the U.S., Fortune 500 companies such as Blockbuster and Countrywide have fallen apart because of their focus on the short-term. Blockbuster in failing to realize that its users were moving rapidly toward streaming video and away from renting DVDs at brick and mortar stores, and Countrywide in making a short-term profit by giving out predatory loans, which would soon stop being paid.
This is a potential threat that will transcend a variety of sectors. There are any number of long-term benefits to education, but they’re ambiguous, hard to prove, and accumulate over an extended period of time. Big data may not help quantify that right away. Exercise improves a variety of health metrics over time—but in a world where everything is measured, it’s easy to imagine that most medical interventions will continue to focus on hospital treatments and medications that yield more immediate results, rather than on long-term, more nebulous investments (such as public parks where people can exercise) that create vague but substantial long-term benefits.
The opportunity, of course, in a world of big data, is that we’ll be increasingly able to not just measure short-term effects, but track and understand the effects of longer-term initiatives. The challenge will be to keep ourselves from acting for the short-term, in the face of so many measurements.