Case Study on How Not to Design a Data Driven Experiment

During my 7 year tenure at NTEN, I’ve had the pleasure of looking at a lot of data, and figuring out ways to use that data in strategic ways to drive the organization and our community forward.

In late spring of 2012, one thing that had been on my radar for awhile was an interesting correlation between our membership renewal rates and event registrations. Specifically, members who had registered for at least one NTEN event over the course of the year were 37% more likely to renew.

After presenting this finding to staff, my hypothesis was that if we were able to increase event attendance, we should see an increase to our struggling renewal rate. This seemed reasonable enough to everyone, so as an experiment we decided to try making all our summer programming free to members in order to increase attendance.

Going into the experiment, I already had a few reservations about our methodology and how exactly we were going to track the results, but momentum pushed us on with the thought that we could sort out those details once we had all the data in front of us. As they’d say on “How I Met Your Mother”, that was a problem for “Future NTEN”.

Jumping forward three months when it was time to actually analyze the data, it quickly became clear that our poor methodology and lack of planning had doomed us to a quagmire of inconclusive results, not to mention any lost staff time or webinar revenue. Of course, it wasn’t a complete loss as we did learn several valuable lessons in how not to design a data driven experiment.

Specifically, my top 5 takeaways from the experience are:

  1. Don’t change too many variables: While we did actually see a jump in attendance for those free events, we forgot to account for the fact that our event calendar that summer and the rest of the year was vastly different than any previous year, meaning there were already far too many variables in play for us to see what affect our “free events” change had actually had. There was also the issue that our renewal numbers are based on a full year of data, while this experiment only ran for 3 months, adding a further layer of difficulty to any analysis.
  2. Setup a control case: In addition to dealing with too many variables, we also had no way of telling what would have happened had we not made those events free. This meant that even if our results had shown a clear shift to support or disprove our hypothesis, there still would have been a question of whether that shift was a result of our experiment or just a random change that would have occurred regardless of what we had done.
  3. Plan out the full experiment ahead of time, including the analysis: We likely would have foreseen many of these issues had we made a plan for the exact data we’d be looking at after the experiment, and how that data was going to help us prove or disprove our hypothesis. Unfortunately by not doing this work up front, we instead ended up with a lot of cloudy data that just raised several new questions instead of answering the one we were asking.
  4. Start with small, easy to design experiments: This was by far the largest data driven experiment NTEN had tried to date, and our lack of experience clearly showed. Looking forward, our new plan has been to hone our skills with smaller, easier to design experiments, and build a foundation of experience that will eventually allow us to explore these larger and more comprehensive strategic questions.
  5. Double check your plan against the scientific method: As a physics major in college, the scientific method was well ingrained in me at that time. However, as this failed experiment plainly demonstrates, the 10+ intervening years have somewhat lessened it’s hold on me. Now while I’m not suggesting you incorporate strict double blind testing for every website A/B test you conduct going forward, it is still worth re-familiarizing yourself with the scientific method concepts in order to catch any major flaws in your experimental plan.

So with any luck, my next blog post on this topic will be about “Future NTEN’s” successes with data driven experiments, but in the meantime hopefully you can benefit from these lessons we learned the hard way.

Karl Hedstrom
IT Director
Karl believes strongly in the power of technology to make our world a better place. While serving as a Peace Corps Volunteer in Niger (’04-’06), he experienced first hand many of the positive impacts cell phones and other newly introduced technologies were starting to bring to his rural village and the entire development sector in West Africa. Following his service, he was inspired to find a career at the emerging intersection of technology and social good, and had the good fortune to join the NTEN team as an AmeriCorps Volunteer in late 2006. For over a decade now, Karl has been working behind the scenes to improve NTEN’s use of technology to effectively meet its mission, and in effect striving to support all nonprofits in doing the same. Karl earned a BA in physics from Pomona College, and now lives in Seattle with his wife, two sons, yellow lab, and a pair of ragdoll cats who can often be seen sleeping on cat shelves in the background of his video calls. Outside of work you’ll find him playing volleyball, disc golf, Pokemon Go, or hiking and exploring the amazing Pacific Northwest with his family.