Email A/B Testing Best Practices Explained

Quoted on this wiki

Every place a page here uses this source, in the order the words come in it.

1. Most marketers are impressed by (and sold on) a feature where you can run a test using a small sample of your full email list. It runs the test until one of the two variants hits a majority in terms of, say, open rate, and then it declares that email to be the winner of the A/B test. At this point, it auto-deploys the winning email to the rest of your list. Sounds simple enough, but from a statistical point of view, this is a terrible idea. “The problem is a lack of statistical significance.” Simply put, statistical significance is the likelihood that an experiment is not due to random chance, entirely meaningless or just flat out wrong.

   In Subject line testing

2. Simply put, statistical significance is the likelihood that an experiment is not due to random chance, entirely meaningless or just flat out wrong. “To achieve an acceptable level of statistical significance, a data set needs to be large enough, and generally the smaller the effect being measured, the larger the data set needs to be.” Statistical significance is measured in terms of % confidence or what’s called a p-value. So, let’s say you want to be 95% sure that the change you’re making to your emails will have a positive effect. Then, you’d want to see test results that show a 95% confidence level or a p-value of 5%.

   In Subject line testing

3. The problem is a lack of statistical significance. You might be asking, “statistical what?” “Simply put, statistical significance is the likelihood that an experiment is not due to random chance, entirely meaningless or just flat out wrong.” To achieve an acceptable level of statistical significance, a data set needs to be large enough, and generally the smaller the effect being measured, the larger the data set needs to be.

   In Subject line testing

4. “Sounds simple enough, but from a statistical point of view, this is a terrible idea.” The problem is a lack of statistical significance. You might be asking, “statistical what?”

   In Subject line testing

5. “It runs the test until one of the two variants hits a majority in terms of, say, open rate, and then it declares that email to be the winner of the A/B test.” The problem is a lack of statistical significance. You might be asking, “statistical what?”

   In Campaign variant testing

6. It might seem strange that an ESP might actively promote this kind of flawed testing. Most ESPs and marketing automation vendors are not focused on providing the best email testing tools. Generally speaking, their clients have small email lists. Those clients have heard that email marketing is important and as a result, it’s pretty easy to sell them on an email testing product — most don’t even know that they should be looking for statistical significance.As long as the software generates a “winning” and “losing” result, most of their users will accept those results as meaningful, because what they’re really looking for is any result. “Think of it this way: When you’re a marketer performing A/B tests, what you’ve been tasked with generating is insight.” So, most of these software vendors know that any result is better than no result (because you’ll feel like you’re making progress). And, they’re banking on the fact that most marketers aren’t trained in statistics. Net result? It’s like you’re going to a casino where you always win, so you keep coming back.

   In Campaign variant testing

Quotes are verbatim and belong to their publishers. Where a quote has been translated, the original wording is kept below it.