Testing Is Cheaper Than Getting It Wrong!

Submitted on Mon, 2/11/2013 - 5:19pm
Testing is cheaper than getting it wrong! Be sure to always be testing to get the absolute best response from your online community.

Year in, year out, nonprofits are using every available online tactic and tool to make real change. In fact, nonprofits are often on the cutting edge of web technology, using new tools and tactics always looking for new ways to build support for their movements and to cultivate and convert those supporters into donors. But not every new tactic works, and it's critical that nonprofits have short feedback loops to figure out what's working and where to allocate resources.

A critical component in this is a kick-ass testing strategy–a simple, quick, and free mechanism of obtaining qualitative feedback on what's working. Testing removes your biases and guesswork to deliver a set of results which will improve online engagement and convert your audience from leads to dedicated action takers. Numbers don't lie!

Real-time testing (10-10-80), Long-term learning (A-B Split), and Multivariate (a combination of elements) are three types of testing you can you use to generate tangible results. When employing any of these testing frameworks, it is of the utmost importance that your test proves to be statistically significant, meaning that the results reflect a pattern, rather than chance.

Comic by xkcd.

What Should You Test?

Email Open and Click-through Rates

Email messaging is one of the most important avenues of connecting to supporters–you'll want to make sure you get this right. A number of factors can affect your email open rates and subsequently, your email click through rates. Testing these factors will make your emails more effective.

Email Open Rates: You have just a small bit of real estate in your audience's inbox with the goal to maximize the probability that any one user will click on that email and ultimately become an action taker or donor.

  1. Subject Line
  2. Timing of Delivery (day of the week, time of day)
  3. Sender Name/Email Address (personal name vs. donotreply@yourdomain.org)
  4. Preview Text (not all email clients display preview text but this can be a variable)

EXAMPLE: Here's a test that Change.org conducted to maximize the open rate on an email being sent to users who might be interested in a petition started by another user.

It was decided that for this email, variations on email subject line would be tested:

The test email was sent to sample audiences of 7000 to evaluate open rates for the following subject lines:

  1. Freddie Mac Sold Us A Meth Lab
  2. I accidentally Bought A Meth Lab
  3. My 2-year-old Lived In A Meth Lab

Which performed better?

The subject line "I Accidentally Bought A Meth Lab" had the highest open and click through rate! We confirmed the test was statistically significant. Now we can roll out this subject line to the rest of our list.

Email Click-through Rates (CTR): Once a user decides to open an email, many factors can come into play that may increase (or decrease) that person's likelihood to click on certain buttons or continue on to visit your website. Some of the biggest factors include:

  1. Layout (single vs. multiple columns, number of images, amount of text)
  2. Image (size, type)
  3. Button (color, size, placement, quantity)
  4. Personalization
  5. Messaging

You'll want to test these factors individually to identify the best practices to get your audience a-clickin'!

Website Forms

Signup and donation forms are critical sections of your website, where people actively choose to be included in your network. Thus, web form completion rates are one of the most important metrics to track and test. These some variables to consider in your testing:

  1. Number of Fields (required, optional)
  2. Layout
  3. Button (color, placement, messaging)
  4. Image (size, placement, type)
  5. Copy and Messaging

Depending on the performance of your forms before testing, optimizing these variables through repeated testing could result in significant increases of form submissions–and donations!

Segments of Your Audience

Our third area to test is the differentiating subsets of your audience. Each and every user is going to have a differing backgrounds, interests, and reasons for engaging with your organization. It would be wonderful if we could send a personal message to each and every action taker but since you may have thousands of users, we rely on segmentation to define broad groups of users to optimize engagement. A few possible user segments to test:

  1. New vs. Existing Users (welcome messaging, action alerts)
  2. Donor Activity
  3. Geography (personalize based on location)
  4. Acquisition Source

EXAMPLE: Let's refer to this international aid organization who used long term testing based on their acquisition sources to determine which interest group had the highest performance and return

This organization ran four petitions on Change.org that focussed on four different issues (drinking water, child mortality, pediatric AIDS, famine):

They tested and analyzed the data and outcomes associated with the action takers from each issue-based petition. Over the long term, the organization found that those supporters who originally signed the "Drinking Water" and "Child Mortality" petitions proved to be the most active and engaged. This knowledge allowed the organization to better focus future recruitment efforts.

ABT: Always Be Testing

Testing is cheaper than getting it wrong! So be sure to always be testing to get the absolute best response from your online community. Once you've decided what to test, walk through this handy testing checklist to make sure your tests will return meaningful, actionable data:

  1. What am I testing?
  2. What is the goal of this test?
  3. Will what I'm testing (control vs. test) get me to my goal?
  4. How large are my test panels (or groups)?
  5. How many times/how long will I need to conduct the test?
  6. How will I analyze my results?

Finally, once you've completed a round of tests, be sure to learn from your data! It's not enough to test, you need apply your results to obtain the optimal results each and every time. Of course, testing should be an ongoing part of your online efforts. For more information about online testing, including a deeper dive on statistical significance, check out Change.org's Online Testing Resource Guide. And remember, always be testing!

Major credit for this article due to Adelaide Belk and Matt Fender, along with the rest of the amazing Client Management team at Change.org, as well as our good friend and testing guru Brenna Holmes.

Lawrence Grodeska is a web marketing expert with more than 12 years experience leveraging technology and new media to promote programs and campaigns for public education, cause marketing and advocacy. Lawrence serves as the Nonprofit Community Manager at Change.org, where he teaches organizations how to best leverage the world's fastest-growing social action platform to achieve real change.