baltimore Buildings
baltimore Buildings
baltimore Buildings
20NTC Session

A/B Testing: Mistakes Everyone Makes (And How to Avoid Them)


Between the two of us, we’ve seen nearly everything that can go wrong in an A/B test go wrong. Many of these failures resulted in our greatest learnings about how to A/B testing correctly. Let us share with you what to look out for so you don’t have to make some of these mistakes yourself! During this 30 minute session, we will touch on what makes a good test hypothesis, controlling variables, audience size and composition, and evaluating test results — with examples along the way of real live tests gone wrong and explanations of what could have saved them.

Target Audience

Those who have been A/B testing emails, online forms, SMS, or other assets for less than a year


Karen Hopper

Senior Data Strategist


Karen Hopper, Senior Data Strategist, serves as the strategic lead for website and donation form optimization at M+R. In this role, she leads the testing team and develops long-term strategies to optimize websites to recruit more supporters, gather more petition signatures, and raise more money. She relentlessly pursues and proves best practices at every step of the donor journey through rigorous A/B and multivariate testing using tools like Optimizely and Google Optimize, and has experience running successful tests in every major CRM. She goes beyond iterative testing, combining her background in social science with her skills in Google Analytics and optimization platforms to develop data-driven persona and behavior profiles that inform donor journeys and maximize conversions from every audience. Before joining M+R in 2012, she worked in policy and advocacy at National Council of La Raza, Center for Science in the Public Interest, and on Capitol Hill. Karen is a proud alumna of St. Olaf College, where she earned a degree in Political Science and Music.

Ellen Freytag


Planned Parenthood Federation of America

Ellen has worked with the Digital Fundraising and Supporter Engagement team at Planned Parenthood in various capacities over the past 12 years, most recently as Testing and Optimization Manager, and Front-End Developer and Production Manager. They have planned, executed, and analyzed results of Planned Parenthood’s A/B tests for several years — across donation forms, other web forms, email, text messaging, and other digital assets. Ellen is proud to have built and launched more emails, donation forms, and advocacy forms for Planned Parenthood than any other person on this planet (thousands of them). They also help train other staff and consultants on A/B testing, CRMs, HTML, CSS, JS, Mobile Commons, and other tools.





March 25, 2020


1:00pm – 1:30pm



Collaborative Notes

Click here

Session Evaluation

Click here