Assessing the statistical significance of repeated A/B tests with meta-analysis
Abstract
This paper presents the steps to conduct a meta-analysis of a set of repeated A/B tests. Repeated A/B testing is common in professional marketing practice. Each test, however, can only be individually evaluated for statistical significance when using basic statistical analysis. A meta-analysis of the same A/B tests provides three useful analytics outputs: an overall treatment effect across all tests, a confidence interval for that effect to assess statistical significance, and a measure of heterogeneity between tests to assess the context variation between campaigns. Meta-analysis is a useful addition to marketing analytics practice.
The full article is available to subscribers to the journal.
Author's Biography
David Harman is an Assistant Professor in the Department of Marketing at the Opus College of Business, University of St. Thomas, St. Paul. His research focuses on customer engagement and customer retention as well as bringing existing statistical methods into professional marketing analytics practice. He previously worked at AT&T Mobility where he managed the targeting strategy for AT&T’s wireless retention direct marketing campaigns. David has a BA in Liberal Arts from St. John’s College, Santa Fe, NM, an MBA in marketing from the University of Washington, Seattle, WA and a PhD in marketing from the University of Iowa.
Citation
Harman, David (2022, March 1). Assessing the statistical significance of repeated A/B tests with meta-analysis. In the Applied Marketing Analytics: The Peer-Reviewed Journal, Volume 7, Issue 4. https://doi.org/10.69554/UTZN6889.
Publications LLP