August 22, 2013

Case Study on How Not to Design a Data Driven Experiment

During my 7 year tenure at NTEN, I’ve had the pleasure of looking at a lot of data, and figuring out ways to use that data in strategic ways to drive the organization and our community forward.

In late spring of 2012, one thing that had been on my radar for awhile was an interesting correlation between our membership renewal rates and event registrations. Specifically, members who had registered for at least one NTEN event over the course of the year were 37% more likely to renew.

After presenting this finding to staff, my hypothesis was that if we were able to increase event attendance, we should see an increase to our struggling renewal rate. This seemed reasonable enough to everyone, so as an experiment we decided to try making all our summer programming free to members in order to increase attendance.

Going into the experiment, I already had a few reservations about our methodology and how exactly we were going to track the results, but momentum pushed us on with the thought that we could sort out those details once we had all the data in front of us. As they’d say on “How I Met Your Mother”, that was a problem for “Future NTEN”.

Jumping forward three months when it was time to actually analyze the data, it quickly became clear that our poor methodology and lack of planning had doomed us to a quagmire of inconclusive results, not to mention any lost staff time or webinar revenue. Of course, it wasn’t a complete loss as we did learn several valuable lessons in how not to design a data driven experiment.

Specifically, my top 5 takeaways from the experience are:

  1. Don’t change too many variables: While we did actually see a jump in attendance for those free events, we forgot to account for the fact that our event calendar that summer and the rest of the year was vastly different than any previous year, meaning there were already far too many variables in play for us to see what affect our “free events” change had actually had. There was also the issue that our renewal numbers are based on a full year of data, while this experiment only ran for 3 months, adding a further layer of difficulty to any analysis.
  2. Setup a control case: In addition to dealing with too many variables, we also had no way of telling what would have happened had we not made those events free. This meant that even if our results had shown a clear shift to support or disprove our hypothesis, there still would have been a question of whether that shift was a result of our experiment or just a random change that would have occurred regardless of what we had done.
  3. Plan out the full experiment ahead of time, including the analysis: We likely would have foreseen many of these issues had we made a plan for the exact data we’d be looking at after the experiment, and how that data was going to help us prove or disprove our hypothesis. Unfortunately by not doing this work up front, we instead ended up with a lot of cloudy data that just raised several new questions instead of answering the one we were asking.
  4. Start with small, easy to design experiments: This was by far the largest data driven experiment NTEN had tried to date, and our lack of experience clearly showed. Looking forward, our new plan has been to hone our skills with smaller, easier to design experiments, and build a foundation of experience that will eventually allow us to explore these larger and more comprehensive strategic questions.
  5. Double check your plan against the scientific method: As a physics major in college, the scientific method was well ingrained in me at that time. However, as this failed experiment plainly demonstrates, the 10+ intervening years have somewhat lessened it’s hold on me. Now while I’m not suggesting you incorporate strict double blind testing for every website A/B test you conduct going forward, it is still worth re-familiarizing yourself with the scientific method concepts in order to catch any major flaws in your experimental plan.

So with any luck, my next blog post on this topic will be about “Future NTEN’s” successes with data driven experiments, but in the meantime hopefully you can benefit from these lessons we learned the hard way.

Like what you're reading?
Sign up to receive the latest articles and updates on nonprofit tech from NTEN and its community of experts.


Karl Hedstrom
Karl joined the NTEN team in 2006 as an AmeriCorps CTC VISTA Volunteer, and then stayed on after accepting a position as NTEN's Data and Systems Manager. After working remotely from Chicago for 3 years, he moved to Seattle in 2011 and took on a more strategic role as IT Director. Before NTEN, Karl served as a Peace Corps Volunteer in Niger, West Africa. His focus was on community development, but he also worked in several other fields including AIDS awareness, environmental education, and small income generation. This experience allowed him to see first hand the huge impact technology can have in the nonprofit and development fields. Karl graduated from Pomona College in 2002 with a BA in Physics, and this technical background nicely complements his development experience in his role at NTEN. Outside of work, Karl keeps himself occupied with a variety of intramural sports leagues and video games, but also jumps at any chance to escape into the wilderness with his wife, two sons, and yellow lab.
Interest Categories: Data
Tags: Best Practices, Data, Strategy