Back to Gartner for Marketers Blog

The Benefits (and Limits) of Relative Measurements in Marketing

By Jason McNellis | March 22, 2019 | 0 Comments

MarketingData and AnalyticsSearch Marketing

When using hand tools in my woodshop, I can achieve tremendous tolerances. With relative ease, I can change the length, width or thickness of a board by one thousandth of an inch (that’s within the thickness range of human hair). This mastery of board dimensioning doesn’t work if I use a ruler—I simply can’t measure the difference between a board that is .750 and .748 inches thick—but I can feel the relative difference in thickness with my fingers.

Much of the natural world is like this; relative numbers are easier to compare than absolute numbers. In other words, I can easily tell which board is thicker, but I need expensive and well-calibrated calipers to measure that difference of .002 inches. Engineers and scientists use the terms precision and accuracy to describe such differences. In market research, the terms reliability and validity convey the same distinction.

A few measurement tools from Jason’s woodshop

The same measurement dichotomy exists for marketing performance. Some marketing questions are relative, like finding the best landing page configuration for a given customer segment. Sure, it would be better to know exactly how many incremental dollars were created from an optimal configuration, but is it worth the effort? After all, it’s relatively easy to know which configuration is best (through A/B testing, for example), but much harder to measure the incremental sales increase that was driven by that preferred landing page. Would that effort be better spent elsewhere, such as optimizing paid search, digital advertising, or direct mail?

Does this mean marketers should throw out their rulers and use their hands? Unfortunately, it’s not that simple. Most marketers need answers using both techniques and will benefit from assessing whether precision or accuracy is more important for each question.

Over the course of this year, I will author several blogs and reports related to marketing measurement. For those eager to learn more about this topic, consider attending my talk next month on “Numbers Are Not Equal, Which Will You Bet Your Marketing Dollars On?”.

For now, let me close with three guidelines:

  1. Use randomized tests as much as possible to measure marketing. Randomization is like a measurement superpower that solves for sampling bias, regression towards mediocrity, seasonality and many other insight obstacles. By randomly allocating customers or visitors into groups you are equally allocating potential sources of bias—effectively nullifying them (assuming sufficient sample size and properly applied statistical tests). When properly randomized, control groups and A/B testing are every marketer’s friend. Yes, there are marketing situations where randomized testing is tough; variation of in-store signage and measuring the impact of a new customer segmentation scheme come to mind. But before you dismiss the methodology, remember you are also saying no to a measurement superpower. In practice this will happen for some marketing activities. When it does make sure you…
  2. Don’t use precise numbers to make accurate decisions. It’s tempting. The numbers are right there, but what do they really mean? For example, rule-based attribution is often precise when comparing performance within a digital channel. But how do you compare that attributed sales number from digital with an incremental sales number from a direct mail coupon program? You can’t. That requires accuracy (made noticeably more difficult because two very different measurement methods were used). Which channel should you invest your next marketing dollar in? You don’t know. Comparing the two numbers you have is worthless, encouraging you to…
  3. Invest more in marketing measures that require accuracy. According to The CMO Survey, most marketing leaders cannot quantitatively say what the long-term impact of their marketing is. This result has been produced across at least 7 surveys!  A clear sign of how hard accurate marketing measurement is (at least for incremental sales). Marketers can get there over time by investing in certain modeling methods and then taking the time to validate and refine them (for one idea click here, subscription required). This need for additional modeling and validation is the cost of giving up on the measurement super power. When randomization and relative measures are not options, it’s time to pull out your expensive and hopefully well-tuned calipers.

Leave a Comment