What Are We Accomplishing and How Do We Know?

Submitted on Tue, 6/5/2012 - 12:39pm
What can the nonprofit community learn from NTEN's own journey toward demonstrating its impact with data?

"What does your organization do?"

That is a question most nonprofit folks can answer pretty easily. We have our elevator pitches ready to go.

"What is your organization accomplishing?"

Now that's a different question, one that's not as simple to answer for most of us. It's also one that is increasingly being asked by potential donors big and small.

The need to better address this second question is part of the impetus behind the currently hot movement toward becoming "data-driven" organizations. Nonprofits are scrambling to find something they can count and, even better, put into a hip infographic. Ideally these data would demonstrate to funders that their organizations are making an impact.

But what to track and count? Which data can help tell an organization's story in a meaningful way? And, moreover, which data can help an organization continually learn and shape its programming for greatest impact?

These are the decisions NTEN was wrestling with when I bumped into Holly a few months ago. She mentioned that NTEN was anxious to start using data to demonstrate the organization's impact in the nonprofit sector. Her goals were three-fold:

  • to make an even more compelling case to funders and prospective members about the value of NTEN;
  • to inform NTEN's own programming decisions; and
  • to walk the talk – to model the types of data-related behavior NTEN had been encouraging its members to pursue.

The timing was great: NTEN had recently completed a strategic planning process and the staff was on board – ready to start counting and crunching. Holly asked whether I could give NTEN a hand identifying which data to track and measure. To which my answer was, "Yes, but first…"

Before collecting and counting anything, we needed to step back and answer two essential sets of questions:

  • What is NTEN working to achieve? What would NTEN's success look like, concretely? How, exactly, do we think NTEN's programming is getting us there?
  • What does NTEN need to learn about its work and its results? What does NTEN want to be able to tell key stakeholders about its results and the work that led to them?

To answer these questions, we walked NTEN through a series of focused conversations among senior staff, within programmatic teams, and with the full staff. The conversations were intense and sometimes heated as staff members wrestled with articulating specifically how each of NTEN's programs help move toward NTEN's goal: a nonprofit sector in which nonprofits' uses of technology are increasing and improving their mission-related outcomes.

This facilitated process generated:

  • A working logic model – a simple graphic representation of the path from NTEN's work to its desired outcomes (see below)
  • A set of guiding evaluation questions
  • My recommendations of data to collect which would address these questions, as well as recommended data collection methods

Now, with this essential groundwork, NTEN is poised to decide which data to track and measure and how to do so. Holly and her team will use the logic model and the organization's guiding questions to prioritize collecting the data that is most useful and meaningful for NTEN.


NTEN's Logic Model (Click for full size.)

The logic model will provide a road map for NTEN to identify the data that can serve as indicators of its progress, from its programming to its desired outcomes. For each step along the logic model, NTEN can begin developing its indicators by asking, "What would it look like?" (The logic model is short-hand; elsewhere we've elaborated upon each step along the way in more detail. For example, we identified which of NTEN's current programs are designed to foster networks, defined "technology champion," and established where within the universe of nonprofits (NPOs) NTEN is poised to create the greatest impact.)

NTEN's priority evaluation questions will guide their efforts further by helping:

  • Identify where to get started. Eventually, data collection and use will be integral to every aspect of NTEN's workflow. For now, though, NTEN will need to decide where to focus its finite resources without overwhelming its staff.
  • Choose which of several indicators for any given step along the logic model are most important to track.
  • Identify additional data pieces that NTEN will want to gather, beyond the basic indicators that the logic model suggests.
  • Identify its qualitative data needs. The straight-forward logic model/indicator framework tends to point toward gathering quantitative data that describe what happened. Qualitative data will also be needed to address questions NTEN has about how and why its programs contribute – or don't – to its desired outcomes.

The value of this process for NTEN goes beyond informing the organization's uses of data:

  • The staff has gained a clearer and richer picture of NTEN's purpose and programs by working together to develop the logic model. By making unstated assumptions explicit, staff members with different tenures, perspectives, and organizational roles now have a shared understanding of where NTEN is headed.
  • Staff members can now think and talk about what NTEN is achieving, rather than simply describing what they do. This shift in outlook toward "keeping an eye on the prize" will not only affect how they describe NTEN to others, it can shape how staff members approach their work planning and even day-to-day responsibilities.
  • NTEN can – and should – draw upon its logic model when:
    • Developing program and individuals' work plans
    • Considering new programming
    • Reviewing and revising existing programming

The logic model is simply a tool, providing a framework for systematically and deliberately reflecting on how NTEN approaches its work. It is a snap-shot representing the organization's strategy for achieving its goals at this point in time. As NTEN learns from its data analysis and evolves its programming, the model will morph as well.

NTEN is on the cusp of conducting some serious reality testing. The logic model reflects how NTEN thinks it is creating change in the world. As it systematically collects and analyzes its data, NTEN will test this theory. The organization may not like all that it learns. Yet, NTEN and the nonprofits that it serves will be better for it as NTEN strategically uses its new knowledge to shape its programming and further its mission.

Amy Luckey has helped progressive philanthropic foundations increase the impact of their funding via strategic program design and evaluation since 2001. An alumna of the early nonprofit tech circuit rider days, Amy ran one of the earliest national online organizing campaigns. She has a keen understanding of the potential for amplifying and accelerating nonprofits' results by effectively employing new and even not-so-new technologies. Amy has assisted The California Endowment, ZeroDivide, TechSoup Global, and Tides' Community Clinic Initiative among others in evaluating and improving their tech-related programming.