How Multivariate Testing Can Boost Your Email Campaign Results by Denise Keller @benchmarkemail

How Multivariate Testing Can Boost Your Email Campaign Results

An A/B split test is essentially an either/or procedure. Sending out a thousand emails with subject line A is compared to another thousand sent with subject line B to see which one drew the best metrics. Given that the average email newsletter can have dozens of specific aspects that can have a measurable impact on the campaign’s results, the process of testing them all though A/B split testing can become an infinite process.

Multivariate testing allows you to transcend the limits of simple A/B split testing by being able to test multiple variable factors at once, or more than two variations for each individual variable.

The art & science of testing multiple variables & values

Not only can you test subject line A vs. B vs. C and so on, but you can also test those subject lines in conjunction with various preheader texts, image positionings, and calls to action. For example, you might have:

Subject line: Check out this week’s sale; Blowout sale this week; 50% off everything this week only

Call to action: Click here; Download the sale brochure now; Get an additional 10% off voucher today only

Image position: Upper right; upper left; middle left

Therefore you are testing three variables with three values each.

Increasing the number of factors requires a mushrooming number of emails

Each multivariate test is different and the addition of variables and values will increase the number of composite emails required. While a multivariate testing of three variables with three values each will require 27 different emails, one with eight variables with four values each will need 65,536 separate emails to be composed and tested.

Multivariate testing of numerically high variables and values must be reserved for the largest email marketers as fully accurate results can only be had in those cases by sending out emails in the millions.

The N*R*C=S Formula

If you are testing six variables with three values each, you will require a total of 729 separate emails. The formula to determine the number of subscribers required to achieve reliable statistical accuracy is:

Number of Emails Composed [N] x Historical Number of Readers For 1 Conversion [R] x Number of Conversions Required For Accuracy [C] = Number of Subscribers Required [S]

Or N*R*C=S

If we assume that you will establish that the minimum number of conversions to these pages to generate the most reliable results is 100, and your conversion rate averages at 5% (one out of 20), you will plug in the numbers as follows:

729 [N] x 20 [R] x 100 [C] = 1,458,000 [S]

This test would only be suitable to a major national email marketer who has 1.5 million email addresses on hand to carry out the multivariate testing.

You don’t need millions of subscribers for reliable multivariate testing

These massive types of multivariate testing are not necessarily obligatory for most email marketers. Smaller subscription lists can be catered to simply by running a multiple number of multivariate testing. If the six variables with three values each test is split into two tests of three variables with three values each, the numbers per test become:

27 [N] x 20 [R] x 100 [C] = 54,000 [S]

Multiplied by two tests, the total number of subscribers required drops to barely over 100,000 which places it within the reach of a far greater number of email marketing list sizes. Even smaller list sizes may be accommodated by simply lowering the [C] factor to a smaller number, such as the statistically legitimate minimum of 25:

27 [N] x 20 [R] x 25 [C] = 13,500 [S]

This lower [C] allows for just 27,000 subscribers for a dual 3×3 test. The results of these smaller tests will not provide the breadth of statistical insight and precise accuracy in every possible permutation that the 1.5 million email test would, but can certainly be of significant validity in the process of assisting any email marketer to make critical determinations about their campaign content.


Multivariate testing can help every email marketer in refining their subscriber approach to maximize results, and thus warrants application to all campaigns, large or small.  Once you master it, this type of testing can give you micro-accurate info on what works for your email campaigns.

Meet the author:



Denise Keller is COO and founding partner of Benchmark Email, one of the world's leading email marketing services for small businesses. Keller served as Vice President of Finance for restaurant chain Market Broiler and CEO of Accor's Ticket Services Division before transferring into the COO position at Benchmark, where she has been instrumental in bringing in numerous small business customers, as well as high-profile clients that include Mercedes Benz, Siemens, and Kaiser Permanente.

Connect with: denisekeller

Share this page:

Share on linkedin
Share on facebook
Share on twitter
Share on email

Connect with us:

To stay in the know and get the best eMail Marketing information that will teach you how to BEE an eMail Marketing Ninja.

Share this page:

Share on linkedin
Share on facebook
Share on twitter
Share on email

to download our
“27 Free Tools Every
eMail Marketer Needs
in Their Toolbox”


Your privacy is our privilege!
We will never rent, sell or share your data. 
You can unsubscribe at any time, but you won’t want to 😃

  • No comments yet.
  • Add a comment

    SUBSCRIBE to our newsletter
    to download our FREE GUIDE
    “27 Free Tools Every
    eMail Marketer Needs in Their Toolbox”

    Download our free guide 27 Free Tools Every eMail Marketer Needs in Their Toolbox

    Download our FREE GUIDE

    Your privacy is our privilege!
    We will never rent, sell or share your data. 
    You can unsubscribe at any time, but you won’t want to 😃

    This website uses cookies to ensure you get the best experience on our website.