Video Bonus: Conversion Rates Meet Distributions with Tim Wilson

July 11, 2018

As we continue to explore video here and there (you can subscribe to our channel on YouTube just in case you want to be ready for a deluge of material, should that ever come to pass), Tim put together a 20-minute explanation of the underlying statistical mechanics of a metric like conversion rate.

Does it sound like a snoozer? Did we mention that he did the whole thing in the format of xkcd.com?! And it was built using R (he is soooo predictable)?!

As a handy reference:

  • At 1.25X speed, you can watch the whole thing in 16 minutes
  • At 1.5X speed, you can watch it in 13:20
  • At 2X speed, you can watch it in 10 minutes

We’d love to hear what you think of it!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

One comment on “Video Bonus: Conversion Rates Meet Distributions with Tim Wilson

  1. Georgi Georgiev Jul 12, 2018

    Hey Tim,

    That’s a really nice try to delve into the inner-workings of the stats and I’m glad my writings were useful to you 🙂

    One addition or clarification, if you’d allow me. The video continuously depicts the two distributions: of the conversion rate of the control and the variant, as if that is what we most often try to measure/estimate in an A/B test. In reality (most of the time) one performs statistical significance tests and builds confidence intervals for the absolute difference between these two conversion rates (delta = pB – pA), not for the conversion rates per se (pA, pB). The two are, of course, directly related, but are not the same thing. The absolute difference has its own distribution with its own mean, variance, etc.

    Going even further, if one wants to make conclusions about percentage lift, then we have a different metric: the relative change (percentage change): delta = (pB – pA) / pA for relative change and delta = (pB – pA) / pA * 100% for percentage change, assuming A is the control.

    Due to the division by pA you get higher variance for % lift than for absolute difference, requiring different computations for statistical significance and confidence intervals, as I’ve recently discussed: http://blog.analytics-toolkit.com/2018/confidence-intervals-p-values-percent-change-relative-difference/ .

    Keep up the good work!

    Georgi

The Digital Analytics Power Hour © 2018