This prize-winning paper describes a novel optimization-based approach to learning from experiments conducted in the world of physical retail. This approach solves a long-standing problem of learning from physical retail experiments when treatment effects are small, the environment is highly noisy and nonstationary, and interference and adherence problems are commonplace. Based on this approach, the team developed a new large-scale experimentation platform called “TestOps,” which has been broadly deployed at Anheuser-Busch InBev (ABI), the world’s largest beer producer. The results have been tremendously impactful and positive: the platform currently runs experiments impacting approximately $135 million in revenue every month at ABI. It regularly assists ABI in uncovering strategies that lead to a 1%-2% rise in sales, which would have gone unnoticed using traditional statistical techniques.
The analytics challenge behind TestOps lies in accurately estimating the effect of an experiment, taking into account (a) the potential presence of arbitrary, highly correlated noise; (b) the ability to handle data corruption; and (c) maximizing statistical power. The Massachusetts Institute of Technology (MIT) team recently made a major advancement in this area by approaching the problem as learning in panels with general intervention patterns and noise, and presenting a robust theoretical solution. TestOps is the first large-scale implementation that generalizes this breakthrough and renders it practical.
The TestOps platform has been live in Mexico for one year, running an average of 42 large-scale experiments per month. Each experiment typically affects around 8,000 test stores with interventions that range from store-specific assortment recommendations to personalized promotions. The platform yields an approximately 100-fold increase in power compared to alternatives. Given its resounding success in Mexico, TestOps is already being scaled out to ABI’s Middle America zone – which includes all of Central America – and a global rollout is expected in the next year.
This work represents a significant pioneering effort in developing experimentation platforms that aid in uncovering effective innovations in noisy retail environments. The concept and framework are flexible enough to be valuable in other domains, such as healthcare delivery and public policymaking, where experimentation faces similar challenges.