Multivariate testing (MVT) is a form of controlled experiment in which multiple page elements are tested simultaneously to find the best combination. While A/B testing compares two complete page versions (one variable changed), multivariate testing isolates multiple elements (headline, image, CTA button) and tests all their combinations simultaneously. For example, testing 2 headlines × 2 images × 2 CTAs creates 8 combinations to evaluate. MVT identifies not just which individual elements perform best, but also which combinations of elements work best together — including interaction effects where two elements amplify each other's performance.
Multivariate testing is powerful but requires substantially more traffic than A/B testing. Because the traffic is split across multiple combinations, each combination receives fewer visitors — so reaching statistical significance takes much longer with low-traffic pages. Most UK businesses should run standard A/B tests until they have exhausted the major single-variable opportunities, then consider MVT for high-traffic pages where interaction effects between elements are important to understand.
When to use multivariate testing vs A/B testing
- Use A/B testing when: traffic is moderate (under 5,000 daily visitors to the page), you have a specific hypothesis about one element, or you are new to experimentation
- Use multivariate testing when: traffic is high (10,000+ daily visitors), you have already optimised individual elements through A/B tests and want to find the best combination, and you have the statistical capacity to detect meaningful differences across many combinations
- Both approaches require: a testing platform (Optimizely, VWO, or AB Tasty for MVT), clear success metrics defined before testing, and commitment to running tests until significance is reached
The practical limitation of MVT is sample size requirements: testing 8 combinations at sufficient sample size to reach 95% statistical confidence may require 50,000+ sessions per page. For most UK SMEs, this level of traffic is only realistic on homepages, major product pages, or checkout pages. For lower-traffic pages, sequential A/B testing of individual elements is more practical and nearly as effective at improving conversion over time.
An interaction effect occurs when the combination of two elements performs differently from what you would predict based on their individual performance. For example, Headline A might outperform Headline B individually, and Image X might outperform Image Y individually — but the combination of Headline B + Image X might outperform the combination of Headline A + Image X. A/B testing would have told you Headline A and Image X are winners, but MVT reveals that the best combination is actually Headline B + Image X. Detecting these interactions is the primary reason to choose MVT over sequential A/B testing.
In theory, you can run sequential tests across different page combinations without a dedicated testing platform, but you lose the simultaneous comparison and statistical rigour that makes MVT valuable. Factors like seasonality, traffic quality changes, and time-of-week effects can confound results when tests are run sequentially rather than simultaneously. For genuine MVT, a dedicated tool is required. Google Optimize offered free MVT before being deprecated; current options for MVT include VWO (paid), Optimizely (enterprise), Convert, and AB Tasty.