Blog post

A/B Testing in Marketing: The Customer’s Always Right

By Anna Maria Virzi | February 08, 2018 | 1 Comment

MarketingContent Marketing and Managementmarketing data and analytics

A/B and multivariate tests are designed to let marketers know what messaging, content, offers and visual elements contribute to a lift in conversions. In effect, these testing techniques let customers signal what variations in messaging, images and other elements on a website or other digital channel led to a conversion, such as registering with a site or making a purchase.

What’s the greatest benefit from A/B and multivariate testing?  “An increased proclivity toward data-driven decision making,” says James Meyers, Gartner principal research analyst. “Marketers gain confidence in delivering customers’ preferred experiences when they have data that supports the decision.  After all, test results are numerical proof that customers prefer one experience over another.”

Surprisingly, many marketing organizations have yet to deploy technology to carry out these tests. Gartner’s Marketing Technology Survey of 206 marketers found that only 33% were regularly using A/B or multivariate testing technology and 28% were in the process of deploying a solution.

AB testing


The challenge? These tests  require a commitment of analytical and creative talent and a discipline to testing variations with content and design. “Expectations from A/B and multivariate tests are often unrealistic, inspired by spurious anecdotes (e.g., “A retailer changed the color of the ‘Buy Now’ button and sales doubled!” says Martin Kihn, research VP and author of the research report, Use A/B and Multivariate Testing to Improve Marketing Programs (subscription required.)

Marketers can avoid these pitfalls. Research Director Jane-Anne Mennella advises marketers who are focused on customer experience to use a blend of user experience, customer experience and marketing analytics, such as funnel analysis, page drop-off rates, to determine where to focus testing. (For best practices, Gartner clients can  see Jane-Anne’s report, Optimize Customer and User Experience Using A/B and Multivariate Testing.)


The Gartner Blog Network provides an opportunity for Gartner analysts to test ideas and move research forward. Because the content posted by Gartner analysts on this site does not undergo our standard editorial review, all comments or opinions expressed hereunder are those of the individual contributors and do not represent the views of Gartner, Inc. or its management.

Leave a Comment

1 Comment

  • Jyoti says:

    A/B testing or split testing can help to figure out the right way for marketers doing anything for their marketing strategies. Marketing professional can avoid any pitfall by implementing ab testing in their strategy. Thanks Anna for sharing this survey. Keep doing good work.