What does Simpson's paradox imply in AB testing?
ab-testing, math, statistics, testing
Solution
It's a little difficult to say without seeing the exact data & the dimensions you are testing, but generally speaking you want to make decisions based on the uncombined data. This article from Microsoft gives a pretty clear example of Simpson's paradox in software testing.
Can you provide a clean example of your combined and uncombined data and a brief summary of the test?
Problem
I am doing A/B testing and I am facing Simpson's paradox in my results (day vs month vs total duration of the test). - Does it mean that my a/b testing is not correct/representative? (Some external factor impacted the testing?) - If it is a sign of problem, what are the directions to follow? Thanks for your great help. Further reading: http://en.wikipedia.org/wiki/Simpson%27s_paradox