A/B and multivariate tests are designed to let marketers know what messaging, content, offers and visual elements contribute to a lift in conversions. In effect, these testing techniques let customers signal what variations in messaging, images and other elements on a website or other digital channel led to a conversion, such as registering with a site or making a purchase.
What’s the greatest benefit from A/B and multivariate testing? “An increased proclivity toward data-driven decision making,” says James Meyers, Gartner principal research analyst. “Marketers gain confidence in delivering customers’ preferred experiences when they have data that supports the decision. After all, test results are numerical proof that customers prefer one experience over another.”
Surprisingly, many marketing organizations have yet to deploy technology to carry out these tests. Gartner’s Marketing Technology Survey of 206 marketers found that only 33% were regularly using A/B or multivariate testing technology and 28% were in the process of deploying a solution.
The challenge? These tests require a commitment of analytical and creative talent and a discipline to testing variations with content and design. “Expectations from A/B and multivariate tests are often unrealistic, inspired by spurious anecdotes (e.g., “A retailer changed the color of the ‘Buy Now’ button and sales doubled!” says Martin Kihn, research VP and author of the research report, Use A/B and Multivariate Testing to Improve Marketing Programs (subscription required.)
Marketers can avoid these pitfalls. Research Director Jane-Anne Mennella advises marketers who are focused on customer experience to use a blend of user experience, customer experience and marketing analytics, such as funnel analysis, page drop-off rates, to determine where to focus testing. (For best practices, Gartner clients can see Jane-Anne’s report, Optimize Customer and User Experience Using A/B and Multivariate Testing.)
The Gartner Blog Network provides an opportunity for Gartner analysts to test ideas and move research forward. Because the content posted by Gartner analysts on this site does not undergo our standard editorial review, all comments or opinions expressed hereunder are those of the individual contributors and do not represent the views of Gartner, Inc. or its management.