Regardless of your personal or organization’s viewpoint on the use of data, you, like many nonprofit professionals, may be asked to serve as an “accidental evaluator:” you may not be trained in evaluation, but you are asked to do it anyway as part of your job.
If you run a client database as part of operations, you might be asked to “pull a few numbers.” If you run your organization’s website, you might be asked to report on actions taken when people visit the site (e.g., did they donate? Volunteer?). If you are a program manager, you might be required to report on your program’s activities for a grant proposal or report. If you are front-line staff, you might be asked to provide client stories that reflect your organization’s impact.
Being an accidental evaluator is not an easy task but it is an essential one, particularly for small nonprofits that may not have the resources to have an experienced evaluator on staff. What can you do if you find yourself in this position? Here are some tips to get you started.
What is program evaluation?
The American Evaluation Association (AEA) commissioned a Task Force of highly-regarded evaluation professionals to answer exactly this question; their response was:
Evaluation is a systematic process to determine merit, worth, value or significance. … Programs and projects of all kinds aspire to make the world a better place. Program evaluation answers questions like: To what extent does the program achieve its goals? How can it be improved? Should it continue? Are the results worth what the program costs? Program evaluators gather and analyze data about what programs are doing and accomplishing to answer these kinds of questions. (American Evaluation Association, What is evaluation?)
Why should nonprofits evaluate their programs?
There are a lot of reasons why you might choose to evaluate your programs, but here are two pragmatic reasons: reporting and operations improvements.
Results from the 2014 Nonprofit Finance Fund State of the Sector survey revealed for 27% of respondents, all of their funders were asking for impact metrics in their reports; for 43%, half or more were. For many accidental evaluators, this is probably the main reason you are in the position in the first place—many funders, including federal, state, local, and private grantors require statements of impact.
As nonprofit professionals, we should strive to provide the best programs possible in the most efficient manner possible, both to ensure the well-being of those we serve and to respect our fiduciary responsibility to all those who provide us funding. By using data and learning from your evaluation process, you can deliver high-quality programs efficiently.
Where should you start?
Here are five places where you can start to learn more about the evaluation planning process:
- Better Evaluation is “an international collaboration to improve evaluation practice and theory by sharing information about options (methods or tools) and approaches.” Their Rainbow Framework helps you plan your evaluation by organizing evaluation tasks into seven clusters: manage, define, frame, describe, understand causes, synthesize, report, and support use.
- Social Innovation Fund’s Evaluation Plan Guidance: A Step-by-Step Guide to Designing a Rigorous Evaluation
- Administration for Children and Families, Office of Planning, Research and Evaluation’s The Program Manager’s Guide to Evaluation (Second Edition)
- W.K. Kellogg Foundation’s Logic Model Development Guide and Evaluation Handbook.
- Western Michigan Evaluation Center’s series of “Evaluation Checklists,” including checklists for Evaluation Plans and Operations and Evaluation Design, both by Daniel Stufflebeam.
Before you embark on your evaluation planning, two general pieces of advice:
(1) Don’t run before you can walk! Many people try to jump to statements about their nonprofit’s impact or outcomes without first ensuring that their foundational data—whom they are serving and how—is in place. Said another way, you should be able to answer with confidence and accuracy (i.e., complete data) “Are we serving whom we intended to serve?” and “What services and how much of those services did participants receive?”
(2) Start small and do that thing well. The Urban Institute recently did a great interview with three evaluation professionals—Isaac Castillo, Tony Fujs, and Daniel Tsin—and all had the same advice: start small and think long-term. Some great advice from Tony:
“My advice is always to start small. The typical mistake that I’ve seen many times is to start by trying to collect everything about everything. And then you have information about nothing—because you can’t process the data, or the cost of collecting the data was underestimated, or you don’t have good-quality data so you can’t say anything.”
Where can I get more help?
Find in-person and virtual communities of practice
The American Evaluation Association (AEA) is an international membership organization for evaluators. As its members span a variety of disciplines, settings, and areas of expertise, members can be part of Topical Interest Groups (TIG), which allow for further community development around specific topics (for example, the Nonprofits and Foundation TIG, of which I am a co-chair). AEA has several welcoming communities of practice available for non-members, if you are not quite ready to join, such as AEA365, a curated daily blog with tips and tricks about evaluation, and the AEA EVALTALK listerv.
In-person communities of practice can also be an invaluable resource for understanding how to engage in evaluation in your nonprofit. To begin, check out the listing of AEA affiliates. If there is not one in your area, or it if is not active, start your own! For example, Crittenon Women’s Union, a Boston (MA) nonprofit, organizes an Outcomes Workgroup that meets quarterly and is open to local nonprofit professionals of all levels of evaluation expertise.
Consider hiring a consultant
An evaluation consultant with expertise in the nonprofit sector can provide you with guidance about how to plan an evaluation, based on your needs and resources available. One way to find a consultant is through AEA’s “Find an Evaluator” listings.
Check out online resources related to evaluation
There are many wonderful, and free, online evaluation resources, such as:
- Eval Central is an aggregator of 57 blogs related to evaluation—rather subscribing to each one individually, you can receive a daily compilation of new blog posts.
- Evaluation Flash Cards: Embedding Evaluative Thinking in Organizational Culture, developed by Michael Quinn Patton and published by the Otto Bremer Foundation, provides 25 flash cards on a variety of evaluation topics pertinent to nonprofits, including helpful resources and key takeaways.
- The Innovation Network, a nonprofit evaluation, research, and consulting firm, produces the State of Evaluation in Nonprofits, which highlights key facts about evaluation in nonprofits, and has a free evaluation resource library.
- The Urban Institute has many resources available for nonprofits, including ones on performance measurement. In addition, they have partnered with Child Trends and Social Solutions to create PerformWell, which “provides measurement tools and practical knowledge that human services professionals can use to manage their programs’ day-to-day performance.”
- The Issue Lab from the Foundation Center provides “free access to thousands of case studies, evaluations, white papers, and issue briefs addressing some of the world’s most pressing social problems.”
- Leap of Reason, specifically the book: Leap of Reason: Managing to Outcomes in an Era of Scarcity.
- NTEN’s resources pertaining to evaluation may be a great place for NTEN members to get help, with peers in similar situations.
- Idealware, a nonprofit that helps other nonprofits make smart software decisions, has developed a series of resources about program evaluation, several of which address the use of technology in evaluation.