Email A/B Test Framework

Use this framework every time you run an email A/B test. It ensures you test one variable at a time, reach statistical significance, and actually implement the findings.

Test Planning

Fill out this section before launching any test.

Field	Details
Test name
Date range
Variable being tested	Subject line / Email length / CTA / Send time / Personalization level / Sender name
Hypothesis	”We believe [change] will [expected outcome] because [reasoning]“
Primary metric	Open rate / Reply rate / Click rate / Meeting booked rate
Secondary metric
Sample size per variant	Minimum 200 per variant for email tests
Test duration	Minimum 5 business days
Owner

Variable Isolation Rules

Only test one variable at a time. If you change two things, you cannot attribute the result to either.

Test Type	What to Change	What to Keep Constant
Subject line	Subject line text only	Body, send time, sender, audience
Email length	Body word count	Subject, CTA, send time, sender, audience
CTA	Call-to-action text and placement	Subject, body content, send time, sender
Send time	Day of week or time of day	Subject, body, CTA, sender, audience
Personalization	Level of personalization (generic vs. custom first line)	Subject, CTA, send time, sender
Sender name	From name (first name vs. full name vs. company)	Subject, body, CTA, send time, audience

Test Design Template

Variant A (Control)

Element	Content
Subject line
Preview text
Body (paste full text)
CTA
Send time
Sender name

Variant B (Test)

Element	Content
Subject line
Preview text
Body (paste full text)
CTA
Send time
Sender name

Sample Size and Significance

Do not call a test based on gut feeling. Use these guidelines.

Total Sample Size (per variant)	Minimum Detectable Difference	Confidence Level
100	~15% relative change	Low (directional only)
200	~10% relative change	Moderate (80% confidence)
500	~5% relative change	High (95% confidence)
1,000+	~3% relative change	Very high (99% confidence)

For outbound emails, aim for at least 200 contacts per variant. For marketing emails to larger lists, aim for 500+.

Results Tracking

Metric	Variant A (Control)	Variant B (Test)	Difference	Significant?
Emails sent
Open rate				Y / N
Reply rate				Y / N
Positive reply rate				Y / N
Click rate				Y / N
Meeting booked rate				Y / N
Unsubscribe rate				Y / N

Decision Framework

Result	Action
Variant B wins with statistical significance	Implement Variant B as the new default. Document the learning.
Variant B wins but not statistically significant	Extend the test if possible. If not, keep Variant A and retest with a larger sample.
No meaningful difference	Keep Variant A (simpler is better when results are tied). Test a different variable.
Variant B loses	Keep Variant A. Document why the hypothesis was wrong.

Test Log

Keep a running log of all tests to build institutional knowledge.

Test #	Date	Variable	Hypothesis	Winner	Lift	Sample Size	Notes
1
2
3
4
5

Testing Priority Order

If you are starting from scratch, test in this order. Each test builds on the previous winner.

1. Subject lines (highest impact on open rates)
2. Email length (short vs. long body)
3. CTA type (question vs. statement, soft vs. direct)
4. Send time (morning vs. afternoon, day of week)
5. Personalization level (generic vs. one personalized line vs. fully custom)
6. Sender name (first name only vs. full name vs. title + name)