Close Enough? A Large-Scale Exploration of Non-experimental Approaches to Advertising Measurement

Randomized controlled trials (RCTs) are the gold standard for measuring advertising effectiveness, but they can be costly and difficult to implement. As a result, firms often rely on non-experimental (observational) methods. In this paper, we use a large-scale “experiment-on-experiments” approach to assess how well a variety of state-of-the-art non-experimental methods approximate the results from RCTs. Using data from 25 advertising experiments conducted on Facebook, we compare the estimates from the non-experimental methods to the experimental benchmarks. We find that while some methods perform better than others, no single non-experimental method consistently produces estimates that are “close enough” to the experimental results. The performance of the methods varies substantially across experiments, and the degree of bias is often large and difficult to predict. Our findings caution against the naive use of non-experimental methods for advertising measurement and underscore the value of RCTs.

Recommended citation: Gordon, B. R., Moakler, R., & Zettelmeyer, F. (2023). "Close Enough? A Large-Scale Exploration of Non-experimental Approaches to Advertising Measurement." Marketing Science. 42(4), 768-793.
Download Paper