A/B testing: The importance of significance and test duration

Abstract

This paper demonstrates that various psychological patterns and a lack of knowledge of basic statistical principles can lead to erroneous interpretations of A/B tests. It explains the statistics behind testing with emphasis on the pitfalls to be avoided. Readers will gain an understanding of the importance of significance in A/B testing, but also learn that a significant test is no guarantee of a universally valid outcome. For reliable results, experimenters must also select a representative time frame. Finally, the paper discusses methods to create a healthy testing culture.