Back to Gartner for Marketers Blog

Use Marketing Holdout Tests to Ignite Your Testing Objectives

By James Meyers | August 30, 2018 | 0 Comments

MarketingData and AnalyticsMarketing Organization and Operations

How would you grade your organization’s testing program? How about if I told you that companies like Amazon, Microsoft and Facebook conduct more than 10,000 digital experiments per year? What if I told you that 10,000 was not an aggregate count across them, but an individual count for each of them? Does your org measure up?

Why Test?

High-performing organizations prioritize testing and optimization (see “Survey Analysis: High-Performing Organizations Make Testing, Experimentation and Analytics a Data-Driven Marketing Priority” – Gartner subscription required). They embed tests within their day-to-day marketing, not just in digital marketing channels, but within offline channels too.

Marketers view testing as a way to create competitive advantage in their customer experiences. They also use it as a trusted method to determine incremental sales from marketing.

What Prevents Inception?

The problem is that numerous organizations believe they have to wait until they are analytically mature or properly funded until they initiate tests. They fear a dedicated marketing analytics team is required, or that they need a formal testing program.

Neither of those are true.

Instead, testing is cheap. It doesn’t require fancy software. It doesn’t require fancy skill sets. And it doesn’t require a formal initiative.

What usually holds organizations back is this:  knowing where to start.

Solution: Holdout Tests

The good news is that there is one type of test that brings tremendous value and is also easy to set up. It’s called a holdout test, and it simply requires suppressing a group of people from receiving content.

Don’t have customer-level data? No worries. Audiences can be created either at the customer level — where known customers are placed in either a test or control group — or they can be created at the market level— where markets are placed in either test or control groups. This means that even if you’re a data-deprived B2B2C organization, you have no excuse not to test.

What are the five steps to get started with holdout testing?

  1. Determine your business objective (leads, email signups, sales, etc.) and marketing channel (email, display, paid search, social, etc.)
  2. Identify your target audience (either a group of markets or a population of customers)
  3. Place them into test and control groups, and confirm the groups’ likeness (demographics, climates, lifecycle phase, etc.)
    1. If you don’t have enough data to confirm homogeneity across your test and control groups, run larger populations that minimize potential for bias (greater than 50 markets, or greater than 1,000 customers).
  4. Execute the test and allow time for significant population response
    1. Usually this will be a minimum of one week, but could be as long as 30-90 days
  5. Measure incremental response between groups (using statistical significance), share results, and brainstorm next tests
    1. Pro tip: perform segment-level deep-dives if the two groups behaved too similarly to gain any initial performance insights

Already getting maximum value from holdout tests? Consider A/B, multivariate, bandit or evolutionary algorithms tests to take your testing to the next level (see “Optimize Customer and User Experience Using A/B and Multivariate Testing” – Gartner subscription required).

Wrapping Up

Tell me what you think. Are you running holdout tests today? Are they allowing you to determine channel ROI or another type of benefit? Use the comments section below, and I’ll do my best to respond.

Leave a Comment