(Bonus) 1:1 with Chad Sanderson: The Pitfalls of A/B Testing

May 1, 2018

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

3 comments on “(Bonus) 1:1 with Chad Sanderson: The Pitfalls of A/B Testing

  1. Thanks for the shout-out, Chad! Kudos for tackling these difficult topics.

    I’d add that to understand type I errors and what “avoiding them” means, one needs to understand the difference between nominal significance and actual significance.

    This will lead you to understanding that avoiding type I error does not make sense. It will always be present and what we do with a proper significance calculation (or confidence interval one) is to measure it, to quantify it. Most issues come from improper application of the statistical tools, resulting in nominal significance being different from the actual one. So a tool might spew out “99% significance” when the actual significance is 90%, often due to misapplication of the tool by the user.

The Digital Analytics Power Hour © 2020