Article
The Minimum Viable Creative Testing Process
growthmentor.com
Quoted on this wiki
Every place a page here uses this source, in the order the words come in it.
3. Decide What We Really Need To Learn “Separate your creatives into concepts, themes, and variants, and save our valuable split test slots primarily for new concepts.” For example if one of the new images we made was of a new product line or feature, it may be existentially important to test it. Whereas the ad copy may be small variations on existing themes, and we don’t have to be rigorously scientific. We also shouldn’t care that much about one audience over another, as the best ad for one is usually the best for all. If we eliminate all but 4 of our variations (2 images x 2 ad copies = 4 variations), and update that figure in the “Number of variations/combinations” field, we’re down to 3 and ½ years.
Let’s take a look at an example and you’ll see what I mean. Whenever you plan to run an experiment, it is important to calculate how long it needs to run to reach statistical significance. If you Google for “test duration calculator”, you’ll find a tool like VWO’s Test Duration Calculator, where you can just plug the numbers in, without the need for knowing the formula. “Testing each one of these in combination will give you 30 test variants.” 3 Images x 5 Adcopies x 2 Audiences = 30 Variants
3. Decide What We Really Need To Learn “Not every ad copy will work with every image, so it’s likely we can eliminate a few variations before we start.” For example if one of the new images we made was of a new product line or feature, it may be existentially important to test it. Whereas the ad copy may be small variations on existing themes, and we don’t have to be rigorously scientific. We also shouldn’t care that much about one audience over another, as the best ad for one is usually the best for all. If we eliminate all but 4 of our variations (2 images x 2 ad copies = 4 variations), and update that figure in the “Number of variations/combinations” field, we’re down to 3 and ½ years.
We said at the beginning that statistical significance is a function of how much data you have, but also how many variations you’re testing, and the potential size of the impact of a successful test. Now that we’ve dropped as many variations as we can, and we’re only testing really big changes, the only thing left is increasing the amount of data. We’ve already doubled our budget so there’s no more wiggle room there. However we can change the metric we’re optimizing to. “However there might be an action further up the funnel, say a registration step, or adding a product to cart, that isn’t a full conversion but is a good indicator of a successful visit.” Related: Learn more about funnel analysis to optimize growth by Nick Schwinghamer
Using Test Duration Calculators To Ensure Statistical Significance “Whenever you plan to run an experiment, it is important to calculate how long it needs to run to reach statistical significance.” Let’s imagine we’re a small startup trying to improve performance in our Meta account through creative testing. We worked with a designer and copywriter on our creative strategy, and managed to produce 3 new image variations and 5 new text variations, for the 2 main audiences we’re targeting. Testing each one of these in combination will give you 30 test variants.
Using Test Duration Calculators To Ensure Statistical Significance “If you Google for “test duration calculator”, you’ll find a tool like VWO’s Test Duration Calculator, where you can just plug the numbers in, without the need for knowing the formula.” Let’s imagine we’re a small startup trying to improve performance in our Meta account through creative testing. We worked with a designer and copywriter on our creative strategy, and managed to produce 3 new image variations and 5 new text variations, for the 2 main audiences we’re targeting. Testing each one of these in combination will give you 30 test variants.
“If you ask an expert, the reason they’ll give you is that you need a lot of data before your experiments become statistically significant.” However the vast majority of advertisers are small. Meta makes over $100 billion in advertising revenue, but if you divide that by 10 million active advertisers, it works out at $10,000 per year, per advertiser. If you’re closer to $10k than $1m, you may be wondering how you go from small to big, when you can’t test and learn what works?
However the vast majority of advertisers are small. Meta makes over $100 billion in advertising revenue, but if you divide that by 10 million active advertisers, it works out at $10,000 per year, per advertiser. If you’re closer to $10k than $1m, you may be wondering how you go from small to big, when you can’t test and learn what works? “The key to beating the limitations of statistical significance is understanding that it’s not just a function of how much data you have.” Using Test Duration Calculators To Ensure Statistical Significance
However the vast majority of advertisers are small. Meta makes over $100 billion in advertising revenue, but if you divide that by 10 million active advertisers, it works out at $10,000 per year, per advertiser. If you’re closer to $10k than $1m, you may be wondering how you go from small to big, when you can’t test and learn what works? “The number of variations of creative you’re testing, and the potential size of the impact of a successful test matter as well.” Using Test Duration Calculators To Ensure Statistical Significance