Efficient experimentation: A review of four methods to reduce the costs of A/B testing

Abstract

Thanks to its ability to detect causality, A/B testing has become the standard method for identifying superior variants of advertisements or product features. At the same time, however, it has also become an obstacle to implementation, due to its reliance on large sample sizes and test durations to identify significant effects. Approaches such as stratification, CUPED (Controlled-Experiment using Pre-Experiment Data), p-value adjustment and multi-armed bandits have been developed to mitigate these issues, but managers still struggle to understand them and to select the best methods for established experimentation platforms. This paper addresses this problem by providing a comprehensive overview of selection criteria for these methods and a decision framework to assist managers in selecting the most appropriate approach for their specific situation to reduce time and opportunity costs. While stratification proves advantageous for reducing sample and time requirements in settings affecting primarily new users, CUPED benefits most in time and sample size constrained settings, affecting existing users. If time and sample size constraints persist despite the regular utilisation of these methods, p-value threshold adjustments can be contemplated at the organisational level. Multiarmed bandits have proven most suitable when immediate optimisation is imperative and relevance would have already passed, before a superior version could have been reliably identified using other methods. In conclusion, it is important to consider the organisational requirements as well as the specific task at hand when choosing an experimental method. This article is also included in The Business & Management Collection which can be accessed at https://hstalks.com/business/.