April 14, 2014

Last-Mile Impact Measurement: Mobile is the Way

Why use scissors to cut the grass when you can use a lawn mower? In the world of impact measurement with remote beneficiaries, mobile is the new lawn mower.

Four nonprofits are leading the way in leveraging mobile technology to hear directly from remote beneficiaries. Each has a very different mission:

But all have the shared goal of hearing directly from their beneficiaries about whether their work is working, and mobile enables them to do it.

Where there is no internet

cisco_foundation_main_images-44.jpgIn India, where these nonprofits work, only 137 million households have internet access, out of a population of 1.2 billion. So surveying your beneficiaries online is not an option.

Until recently, organizations used the crude tools of pen-and-paper surveys (challenging with a 65% female literacy rate) or in-person interviews (not reliable for sensitive questions as the respondent may just tell you what they think you want to hear). Neither of these is scalable, and both are error-prone since they require manual data entry.

Deploying pen-and-paper surveys with a low literacy population is a bit like mowing the lawn with scissors. It’s time-consuming and doesn’t quite get the job done.

Yet in the same country there are 929 million mobile subscribers and growing (soon approaching the total population size). Now we’re onto something.

How it works

cisco_foundation_main_images-31.jpgWe designed the Labor Link platform to help organizations leverage this new mobile connectivity to listen to beneficiaries. We made it free, anonymous, and voice-based (not SMS) – so it does not require literacy. It also works on basic feature phones, the medium most familiar to the masses.

Target respondents call a local phone number, place a missed call (let it ring and hang up, a common practice in India), and immediately get a call-back from our automated system. They answer 10-12 multiple-choice questions with their touchtone keypad that are voice-recorded in Hindi, Tamil, or other local language.

Then we analyze that data for the nonprofit, giving them real-time feedback from the field.  And close the loop with the respondent through voice-recorded messages that make them feel heard and share locally relevant educational content.

How nonprofits are using it

Following a high-profile rape case in 2012 in New Delhi that mobilized advocates for women’s rights and safety, SAI, for example, wanted to measure workers’ understanding of their right to equal treatment in the workplace. The organization was running a training program on gender discrimination in garment factories. The project was funded by DFID, the UK international aid agency, as part of a broader effort called Responsible and Accountable Garment Sector (RAGS) to improve working conditions in garment factories.

We partnered with SAI to design a survey on women’s issues and delivered it at seven factories employing over 12,000 people. It asked questions like “Is rude language sometimes necessary to communicate urgency in the factory?” and “Are there some jobs at the factory that are only suitable for men or women, but not both?”

The response was overwhelming. Nearly 40% of targeted workers completed surveys, consistent with other survey campaigns we’ve conducted and much higher than the average 5-10% on customer surveys.

Other nonprofits we work with have had a similar experience. VisionSpring is using Labor Link to understand if the people reached by their eyeglass distribution campaigns could afford or had access to other eyeglasses, and the impact of their new glasses on productivity at work or school. GoodWeave is giving rug weavers an anonymous channel to report on sensitive issues like the presence of child labor in their workplace. And Fair Trade USA is measuring the social and economic ripple effect of Fair Trade in communities that grow tea, coffee, and fresh fruit and vegetables.

What we’re learning

Through these partnerships in India, we’ve learned a few things that maximize participation, data quality, and value across the board. To survey beneficiaries effectively via mobile, keep in mind these 5 things:

  1. Keep the survey short. Anything longer than 4 minutes on the phone and callers start to drop off, which affects data quality. That translates into about 10-12 multiple-choice questions.
  2. Offer simple incentives for participation. We typically use mobile credit because it’s a universal currency and can be administered virtually to the respondent’s phone.
  3. Close the loop with respondents. Think of the last survey you took. Do you know what happened to the data? Make sure your respondents are not answering questions into a black hole. Thank them, and let them know what you found and what you’re doing about it.
  4. Compare with other data sources for a 3D picture. What other data sources do you have access to – either your own data or public sources like UN or World Bank data – to provide additional context?
  5. Re-package the findings for partners. We find that maximum transparency is best. Share the results with respondents and other stakeholders, such as shop-keepers (for customer surveys) or employers (for workplace surveys). It builds trust for future information sharing.

Lastly, let’s share with the wider world. This is innovative stuff. We need a community of practice around really listening to beneficiaries and reporting back on what we find. Our nonprofit partners have found that donors are eager for such direct beneficiary feedback. In fact, some of this work in India is supported by a grant from USAID’s Development Innovation Ventures (DIV) unit.

Nonprofits are also using Labor Link mobile survey data to strengthen business relationships. “Labor Link enables farm management to stay intimately connected to their workers,” says Hannah Freeman, Director of Produce & Floral at Fair Trade USA. “In Fair Trade, we see it as an opportunity to strengthen communication and improve the operational efficiency of our key partners.”

Just as there are many ways to cut the lawn – from scissors to a lawn mower, or even a goat – there are many ways to capture impact data. But when you’re trying to reach a low literacy population that lacks internet access, mobile is the way to get it done in a way that’s reliable, affordable, and scalable.

Like what you're reading?
Sign up to receive the latest articles and updates on nonprofit tech from NTEN and its community of experts.

Subscribe

Heather Franzese
Heather Franzese is the Executive Director of Good World Solutions (GWS), the nonprofit social enterprise behind the Labor Link mobile engagement platform. Recognized as a Purpose Economy 100 (PE100) global changemaker, Heather has been working for 15 years to improve the lives of vulnerable workers in global supply chains. She was one of 12 social entrepreneurs in the 2013 accelerator class at Santa Clara University’s Global Social Benefit Institute at the Center for Science, Technology and Society. Since 2010, Labor Link has bridged the gap to over 80,000 workers and farmers in 12 countries, including China, India and Brazil. Heather brings together industry experience managing CSR for Columbia Sportswear Company’s licensed and collegiate categories, launching Fair Trade certification for apparel, raising $4 million in social impact funding, and field experience with small-scale farmers in Peace Corps Mali. She holds a Masters in Economic Development from Harvard Kennedy School. Follow Good World Solutions on Twitter: @GoodWorldTech.