A few months ago, I asked, “Are You an Accidental Evaluator?” If you find yourself in this position, you may soon realize that the amount of data that you need to collect and analyze is daunting. Luckily, there is a lot of available technology to support evaluation, but the process to determine what you need can be overwhelming and potentially very costly if you choose wrong.
Why Technology in Evaluation?
“A good database and reporting tool can greatly ease the process of collecting, recording, and analyzing the output and outcome data an organization is tracking. . . Similarly, the lack of flexible tools can hinder an organization’s effectiveness and add unnecessary time to the evaluation process.” (Idealware, The Reality of Measuring Human Services Programs: Results of a Survey p. 15)
Technology is a tool that can facilitate your organization’s evaluation process, but it is not a magic box. The technology needs to support your evaluation plan; your evaluation plan shouldn’t be dictated by your technology. For more information about evaluation planning, check out the resources I listed in my other piece.
How Can Technology Support Evaluation?
Technology can support evaluation in several keys ways:
- Facilitating data collection (e.g., via online forms)
- Storing data (e.g., on people and activities)
- Analyzing data (i.e., as often done using Excel, SPSS, Stata, R, GIS, etc.)
- Reporting (e.g., dashboards, monitoring reports)
What Types of Software Can Support Evaluation?1
There are many types of software—covering a wide spectrum of quality and cost—that can support your evaluation needs. These may include:
- Custom Databases (e.g., MS Access, Filemaker)
- Constituent Relationship Management (CRM) Software (e.g., CiviCRM, Salesforce)
- Case Management System (e.g., Efforts-to-Outcomes, Apricot)
- Homelessness, Learning, Membership, or Legal Case Management Information Systems
- Electronic Medical Record Software
How Do You Choose Technology for Evaluation?
What is your evaluation scope? That is, how complex are your evaluation needs? The more complex, the more data you will need to gather, monitor, analyze, and report, so your technology needs to be able to support that complexity. This goes back to the importance of evaluation planning. As described in more detail in a recent AEA365 blog post, at Jewish Family & Children’s Service of Boston (JF&CS), Rachel Albert, the Vice President of Learning and Impact, and I have developed a tool for determining the scope of a nonprofit’s internal evaluation needs.
What is your evaluation scale? How large are your evaluation needs, whether it be the number of clients served, programs, or staff members? For example, we serve 17,000 people a year across 42 programs, each with their own evaluation scope. Therefore, our evaluation scale is quite large, and so our system needs to be robust enough to handle that volume of data.
What resources do you have available? By resources, I mean two distinct things. First, I mean pure capital: money for licensing (at initial purchase and ongoing, depending on the contract), implementation (in particular if customization is needed), and support. Second, I mean people resources: in order for an evaluation technology to be successfully implemented in an organization, there needs to be someone in charge of supporting it, whether a staff member or contractor. Additionally, your technology support personnel needs will grow with the scope and scale of your evaluation needs. We have one-and-a-half full-time staff members devoted to database administration. I cannot stress enough the importance of being thoughtful and realistic about what resources you have available to implement a technology for evaluation before making a purchase.
What Features Should You Consider in a Technology for Evaluation?
Determining the scope and scale of your evaluation needs directly informs the features that you will need to consider, and of course the resources you have available will mediate your decision. Some questions you should ask yourself include:
Is the technology cloud-based or local (i.e., installed on individual machines)? If you have multiple people who need to access the system, or if you have staff members who need to be able to access the system from the field, you may need a cloud-based solution. A cloud-based solution can also reduce support costs.
Is the technology customizable? Some technology is acquired “as is:” you have to work within the constraints of the system with little ability to customize. However, other technology will allow for customization (often at a cost), which will allow you to tailor it to your organization’s particular needs. Alternately, some software options will work “out of the box” while others must be customized before use, which may potentially increase the cost significantly.
Does the technology use open standards? Some platforms that support evaluation are “closed systems”—meaning, you must rely on the developers to create new features. Other systems are based on open standards, meaning you can access other applications, often through a marketplace, to enhance your system.
Does the technology support different levels of users? Depending on the scope of your evaluation plan, you may need multiple staff members to be using the system, each with different access permissions.
Does the technology support constituent and/or non-constituent-level data? You may need to store data both about the people you serve and the activities in which they participate. For example, I worked in a nonprofit in which we needed to gather data not only about participants who took our trainings, but also on the training sessions themselves (e.g., when and where they were being held, the trainer, the status of the training planning, etc.).
Is your nonprofit required to use a federal-, state- or grant-mandated database system? If you use an alternate system, will you be able to export your data in order to be able to import it into the required system?
What additional security features are required or desired? For example, does your system need to be HIPAA-compliant? As another example, The National Network to End Domestic Violence has created a resource, focusing on security, to use when selecting a database for domestic violence and sexual assault programs.
Here are some additional considerations:
- Does the technology need to support data collection tools?
- Does the technology need to have analysis or reporting tools?
- Does the technology need to support billing?
- Does all or parts of the technology need to be an approved electronic medial record?
- Is the technology accessible on mobile devices?
If you are trying to choose a technology to support your evaluation needs, here are my key takeaways:
- Remember that technology should be used as a tool to support a thoughtful implementation of an evaluation plan, not a substitute for one.
- Consider the scale and scope of your evaluation needs.
- Consider your available resources.
- Then look at technology options!
Where Can You Get More Help?
If you are looking for more support in choosing a technology for evaluation, following are some helpful resources.
Idealware has several resources about technology and program evaluation, including the reports Software to support program evaluation and Nonprofit performance management: Using data to measure and improve programs.
1 Adapted from: Christian, R., Quinn, L.S., Andrei, K., Pope, E. The Reality of Measuring Human Services Programs: Results of a Survey. Portland, ME: Idealware. Available from: http://www.idealware.org/reports/reality-measuring-human-service-programs-results-survey, p. 15.