Predictive Program Evaluation: Don't React, Predict

Submitted on Tue, 10/2/2012 - 1:01pm
To truly understand the power of a predictive program evaluation solution, let's first discuss three common misconceptions that many nonprofit organizations have.

Any type of program evaluation tends to be reactive: something went right, so we keep on doing it, or something went wrong, so we try to fix it. Reacting is better than not acting at all, but relying solely on the after-the-fact "what happened?" -- often gathered ad-hoc in spreadsheets and anecdotal stories -- fails to take into account the hidden patterns that lie within the data as well as the dynamic factors that accompany reality.

By leveraging all of our available data and analyzing it, we can move from reaction to prediction, uncovering program inhibitors and drivers to predict how our programs will perform and what to do now to make them better. We call this predictive program evaluation, which is fueled by predictive analytics.

To truly understand the power of a predictive program evaluation solution, let's first discuss three common misconceptions that many nonprofit organizations have:

  1. We don't have the data necessary to effectively evaluate our programs.
  2. We cannot gain the deep, granular insight to uncover the key predictors that contribute to success.
  3. We struggle to visualize results of program performance and share them with the key stakeholders and donors we rely on for funding.

By understanding how predictive analytics breaks down these common misconceptions, nonprofit organizations will be able to leverage the data and expertise they already have in-house to deliver successful, efficiently-run programs.

Misconception #1: We don't have the data!

At the very least, nonprofits usually have access to basic information about their constituents: their demographics (name, age, birthplace), their interactional history (when they received services, what services they received, how they contacted the organization), and their transactional history (if/when/how much they donated).

This may be held in a CRM database, or perhaps even just a spreadsheet; regardless, it's data -- and an excellent place to start. By employing easy-to-use, statistical analyses on this data, organizations can begin to understand the types of services a certain segment is utilizing or why a certain program did not meet its objectives.

In order to gain a true, 360-degree view of their constituents, nonprofits can enhance these traditional data types with unstructured data, like surveys and social media.

Surveying is an excellent tool for gaining additional information may not be readily available, particularly in terms of attitudes and preferences. Often times, the extremely valuable survey data comes in open-ended, free-response questions ("How can we better serve you?").

The same is true for social media data -- the true gems can come from meaty blog posts or twitter feeds. Before, these types of unstructured data were only used as "listening" tools, with organizations unable to take the time and manpower to incorporate them into deep analysis. With predictive analytics, however, true sentiment can be extracted from text, putting structure around unstructured data that can be fed into analyses for actionable insight. Through sentiment analysis, for example, nonprofits can monitor public perception of their organization, understanding if a particular service is being talked about in a positive, negative, neutral, or ambivalent tone.

Misconception #2: We don't have the insight!

Usually, it is not just one thing and one thing only that contributes to a program's success or failure. More than likely, it is a combination of factors. The trouble is, nonprofits often don't have the resources to determine what those combinations are -- especially because, in reality, there can be hundreds or thousands of factors to choose from.

For example, Medway Youth Trust, a nonprofit that focuses on improving the quality of life for youth in the United Kingdom, wanted to identify young people at risk of becoming "NEET," -- "not in education, employment, or training" -- an increasing problem for the United Kingdom in terms of developing the nation's youth as well as society as a whole. Based on historical data from records of both NEET and non-NEET individuals, the nonprofit realized there were potentially over 1,000 factors that could affect NEET status, ranging from family history to elements in conversations with youth counselors.

How do you deal with 1,000 different factors when conducting analysis? Particularly problematic, as in the case of Medway Youth Trust, much of the data was unstructured, written up as free text comments and action plans from interviews with young people.

Medway Youth Trust utilized predictive analytics to create a propensity model that incorporated disparate data sources from both NEET and non-NEET individuals, uncovering hidden patterns and associations in the data to score the primary factors that contributed to NEET status. By running new data on school-age children through the model, the nonprofit could predict which of them were most likely to become NEET in the future. Generating a report that listed every individual that had a greater than 60% chance of becoming NEET, Medway was able to focus its resources on intervening with those who needed the most help.

Misconception #3: We can't visualize results!

As we all know well, nonprofits rely on grants and donations to keep them up-and-running. As such, it is absolutely critical for these organizations to be able to track and report their performance to key stakeholders within their organization, donors, and those involved in contributing grants. The first step is ensuring the organization has quality data, as improperly compiled data leads to inaccurate results. For Medway Youth Trust, any discrepancies between their structured and unstructured data were flagged by the predictive analytics' solution for review. Previously, reviewing all records manually took up to a month; with predictive analytics, it was reduced to a few hours.

Next, it's important to be able to have the results of program performance easily consumable via dashboards and scorecards. A potential donor does not want to see a list of 1,000 factors that contribute to NEET; rather, a graph that displays how 723 individuals were proactively identified the previous year as being at risk and were now thriving in education, employment or training, would be significantly more powerful.

Finally, those involved in implementing a predictive program evaluation strategy understand reality: things change, nothing is static, and there are forces beyond our control. By comparing "what if" scenarios, organizations can forecast and plan effectively by taking into account various situations that may occur. For example, what happens if our budget decreases by 15%? How would that affect volunteer engagement, allocation of overhead costs, program services, etc?

By simulating potential scenarios, nonprofits can understand all the factors that would be affected and proactively plan accordingly.

By implementing a predictive program evaluation solution, nonprofit organizations can move from react to predict -- saving time, money, resources, and most of all, driving program success.

Please visit the IBM SPSS Nonprofit Resource Center to learn more, or contact us at 800-543-2185.