Subject line testing asks which wording earns a response from this audience. Use opens to diagnose attention, then judge the campaign by replies.1 Speed is the main trap: a test can look decisive while lacking statistical significance.2 Build clean comparisons, keep the message stable, and wait for the result before changing the rule for the audience.
Set the decision before writing variants
Start by fixing the decision you want the test to inform. This keeps an attractive open rate from becoming the reason you scale wording that produces no conversations.
Judge subject lines by replies at campaign level, using one subject line per campaign.3 The first sentence also acts as preview text and can affect the open rate.4 Deliverability, sender name, time of day, and preheader text also affect the open rate, along with the subject line.5 Fix deliverability and list quality first because they set the ceiling for what the subject line can achieve.6
Write down the audience, the reply you want, and the signal that will tell you to keep testing. Ask, "What would make this subject line worth keeping?" That question gets the team to agree on the result before a promising open rate starts driving the decision.
Build a clean comparison
Build each test so you can explain the result. When more of the email changes at once, the conclusion becomes less useful.
An A/B test sends two subject line variants to comparable segments and compares performance to learn what a specific list responds to.7 Keep the email itself constant while the subject line changes, so both variants carry the same message.8 Test one dimension at a time: length, format, or personalization depth.9
Use plain language and make the subject line specific to something real about the recipient.6 Specific, contextual subject lines outperform generic curiosity oriented subject lines.10 Write the variants before sending and label the angle each one tests. A label such as length or personalization gives the result somewhere useful to land when you review it.
Run the test with enough data
Give the comparison enough time and recipients for the result to mean something. Set the review point before the send so an early lead does not become an unplanned stopping rule.
A smaller effect generally requires a larger data set to measure it with acceptable statistical significance.11 Statistical significance asks whether the result could be explained by random chance.12 Record the audience, variant, open result, and reply result together. Keep the same comparison running until the review point you set, then decide what to carry forward.
If the variants differ by a small wording detail, expect the read to need more data than a test built around a large change. That is a reason to plan the test carefully, not a reason to make several changes at once. When the result is unclear, keep the finding as a question for the next audience segment.
Read opens, then inspect replies
Open data
- If opens are low, A/B test subject lines
- If email is not opened, test it out
Read the result in layers. Open data tells you whether the subject line earned attention, while reply data tells you whether the campaign created a useful next step.
When opens are low but CTOR is high, the content is strong and the subject line or inbox placement is weak; A/B test the subject lines and check authentication.13 If an email is not opened, test a different subject line.14 Keep the body stable while you make that change, then compare the new result with the original audience and campaign context.
A rise in opens does not prove that the wording is working for the business. A subject line can win the first action and still fail the reply test. Use the reply result to decide whether the wording deserves another audience or a place in the campaign pattern.
Turn the result into an audience pattern
A winning line gives you a hypothesis about an audience, not a permanent rule. Store the result by audience and test the pattern again when the segment changes.
Analytics can identify the subject lines that perform best for particular prospect pools, and A/B testing can optimize messaging for that audience.15 Test different lengths with comparable audiences.16 A two to four word question format was associated with an 11 percentage point open rate increase, from 35% to 46%.17 Use that format as a hypothesis to test, since a short subject line approach may not fit every business.18
Use prior email performance data to inform the subject line strategy.19 During review, ask, "What subject lines have performed best in this campaign?".20 Keep the answer attached to the audience and the reply result. No subject line example is guaranteed to work for every audience, so carry patterns forward as testable assumptions.21 Continue to A/B test and refine subject lines. Review analytics to learn what resonates with each specific audience.22
What not to do
Avoid shortcuts that make a subject line test look conclusive while leaving you with a weak decision.