When less is more: Privacy by design in A/B testing

Abstract

Many A/B testing programmes collect, store, share and analyse individual-level customer data, even though the privacy-by-design principle of data minimisation holds that only the minimal data necessary to achieve the intended purpose should be used. This paper provides a method for A/B testing that supports privacy goals through data minimisation. An easy-to-apply solution is presented that relies on K-anonymity — a privacy method for enforcing data minimisation — and ordinary least squares regression. To illustrate the general utility of this approach, two advanced use cases in A/B testing are offered: (1) a method to discover both potential pairwise A/B test interactions and to surface segment-level average treatment effects (conditional ATE); and (2) a covariateadjusted framework for A/B tests based on ANCOVA2 for enhancing test precision and/ or reducing testing time. This article is also included in The Business & Management Collection which can be accessed at https://hstalks.com/business/.