It was 2009, at the height of the financial crisis, and world leaders were struggling to understand how they had been so taken by surprise by an economic shock that had struck their populations with the ferocity and suddenness of a natural disaster.
They were seeking answers to the questions raised by this global economic catastrophe in the United Nations, asking Secretary-General Ban Ki-moon what insights he and the Organization could offer on the impact of the crisis in the many countries where the UN operates.
The speed with which the global economy had been swallowed by the crisis exposed the gaps in governments’ knowledge and understanding of their peoples and revealed the absence of up-to-the-minute, accurate data about who was being affected and to what degree.
All they had were three- to five-year-old household surveys, censuses and other national statistical collections, which could only provide limited guidance on how to respond to the suddenly devastated global economic landscape.
After consulting with UN agencies, Mr. Ban was forced to come to a similar conclusion. The UN had nothing on the scale necessary to close the information gap and to equip governments with data that was updating fast enough to inform effective responses to the sorts of situations they were facing on the ground.
What emerged was UN Global Pulse, a body within the Secretary-General’s office dedicated to tracking development progress and emerging vulnerabilities in real-time by tapping into Big Data – the enormous volumes of records that are automatically created by people’s every interaction with the digital world.
Robert Kirkpatrick, the Director of UN Global Pulse, says Big Data is aptly named, as the volumes are mind-bogglingly huge and continue growing exponentially.
“The amount of information being produced is more than doubling every year,” he says. “In 2014 alone, there was more data produced than in all of history combined.”
The vast majority of that data is held by private sector firms, like telecommunications providers, banks and credit card issuers, and social media companies.
Every time a credit card is swiped, data is created. Every time a mobile phone call is made, a text sent or airtime added, data is created. Every time a status is posted, a ‘check-in’ made or someone’s post ‘liked,’ data is created. The firms then gather this ‘data exhaust’ and analyze the information contained within to support business decision-making.
“Information out there is being used to commercial advantage by these companies,” says Mr. Kirkpatrick. “So they can identify new customers; so they can identify new market trends. They can monitor their own operations in real-time. What hasn’t been done with this data, systematically, to date, is learning how to use it as a public good.”
As well as equipping governments and the UN to respond better to economic disasters like the 2008-9 meltdown, the designation of Global Pulse also contained within it recognition of the fact that achieving the new development agenda would require 21st Century techniques and approaches.
“Everyone has recognized that to achieve sustainable development, we’re going to need to move beyond traditional approaches of using surveys once or twice a year to monitor what’s happening in communities where development work is underway,” says Mr. Kirkpatrick.
“What we see now is that the bulk of the information out there that is relevant to global development is actually being produced by the beneficiaries themselves, not being collected by national statistics offices. So the opportunity here is to use this kind of information both to measure sustainable development in ways not done before but also to achieve it.”
Global Pulse is one of the world’s most innovative responses to the challenge, seeking to harness the power of Big Data to provide immediate and constantly updated insights on issues such as health, education, employment and food security, to support global development and humanitarian action.
In pursuit of that, Global Pulse has established a network of ‘Innovation Labs’ in New York, Kampala and Jakarta, which seek to partner with UN agencies and governments and to experiment with new sources of Big Data to find out how they can be used to close information gaps – either by providing totally new insights or producing information that is already known but in a faster or cheaper way.
Typically, a UN agency will tell the nearest Pulse Lab what problem they are trying to solve and what type of information they need to be able to do so. Global Pulse then brings to bear its network of partners in the private sector and academia, who offer the data and human expertise needed to complete the work.
Once a methodology is in place for estimating an aspect of development through application of Big Data, the Lab can then turn that methodology into an application that can be used by the UN agency on a daily or a weekly basis to better understand how, where and why their development programmes are making a difference or to understand how quickly a population is recovering from a natural disaster.
“The data sources that are useful for development fall into two broad categories: what people say and what they do,” says Mr. Kirkpatrick. “How people spend money on global air-time products predicts how much they’re spending on food and predicts their overall level of poverty. How people move about a city, which you can see through mobile phone networks, predicts how they’re being displaced by a natural disaster. How people tweet the price of food predicts the actual food prices.”
When talking about “what people say,” Mr. Kirkpatrick is referring to information readily available online in digital form publicly – for example on Twitter or Facebook. That information can be analyzed readily to understand what people think and what they feel and how that is changing over time.
The information on “what people do” meanwhile is much less subjective than the information found on social media and is in fact extremely precise because it is produced as a by-product of people’s use of digital services. Some of the results of such studies can be incredibly revealing.
In one project, Global Pulse partnered with the World Food Programme (WFP) in central Africa. Global Pulse was given access to tens of millions of anonymous telephone records which revealed how people spend money on air-time purchases in the country.
The air-time spending patterns were compared to 7,000 WFP household surveys, which measured their consumption of food and placed them in the multidimensional poverty index.
“What we found was astounding,” says Mr. Kirkpatrick. “How people spend money on mobile airtime is an incredibly powerful predictor of their poverty level. WFP does these kinds of surveys every two, three, four years. A mobile network – just as a by-product of providing services to these populations – is producing every second the means to generate an updated understanding of their economic well-being.”
The information gathered from air-time purchase data, because it correlates so closely with information known from surveys and the poverty index provides what Mr. Kirkpatrick calls a “proxy measure” for poverty. The close correlation between the information in the various sources provides the government, the UN and development partners with a real-time technique for tracking poverty.
“That, we believe, has profound implications,” says Mr. Kirkpatrick. “This type of information can be used for policy planning. Unexplained drops in spending patterns on air-time indicate that people also don’t have enough money for food and medicines and education. That means that we now have an early warning system that can be used for much more precise and rapid targeting of response.”
The UN is in a neutral space between governments, communities and the private sector and we’re able to use our labs as a safe space for innovation.
Transforming Big Data into a genuine public good first and foremost requires ensuring a "do no harm approach" – privacy must be protected at every step. To that end, the Global Pulse initiative has a strong focus on data privacy and protection, and works only with data-sets containing no personally identifiable information, per its Data Privacy Principles. Global Pulse also advocates for richer public dialogue and discourse about data privacy in the context of sustainable development, through conducting research on the topic, assembling a global Data Privacy Advisory Group and convening public events.
While some companies remain hesitant about sharing the their data – mainly because of concerns over giving away competitive advantage or because of fears about invasion of privacy – there are also difficulties associated with bringing together practitioners from diverse and otherwise unconnected fields to work on Global Pulse’s projects.
“One of the main challenges that this whole area faces is getting everyone together to do the work,” he says. “Getting together the data science expertise and the development expertise. These communities have not historically been connected to each other and that’s one of the main functions of our Pulse Labs.”
He says Global Pulse has been working with the private sector on an idea called ‘Data Philanthropy,’ whereby their data can be safely and responsibly used in the cause of sustainable development. As a UN body, Global Pulse has found it has some unique advantages in that realm.
“The UN is in a neutral space between governments, communities and the private sector and we’re able to use our labs as a safe space for innovation,” he says. “I think that’s a really important role for Global Pulse – creating that sandbox where they try new things and learn very quickly what avenues we should be following.”
Attention on the field is certainly growing, particularly in light of the Secretary-General’s Independent Expert Advisory Group’s report on the data revolution for sustainable development, called ‘A World That Counts.’ Within the private and public sectors, as well as in civil society, there is now tremendous interest in how new technologies and approaches can transform development and humanitarian operations and ensure the inclusion of all in development processes.
“Big Data is a new digital natural resource and it has tremendous potential to be used to achieve sustainable development and to measure progress,” says Mr. Kirkpatrick. “Data is now an exciting topic in a way that it wasn’t before. And we are very keen to build on that momentum.”