This is in my opinion the best simple A/B test significance calculator. I like it for two reasons:
1) The multiple testing correction, which many people ignore when testing > 1 variants vs. the control
2) They display not only the p-value, but the confidence interval around the _difference_ between the variants (i.e. the effect size). It's tempting to observe an increase of say 50% that is statistically significant and think that you can be confident about the 50%. This is not the case! You chose your sample size to be confident that the actual effect size is > 0 -- not that you would have a tight confidence interval around whatever difference you observed. Depending on your sample, the lower bound could still be close to 0.
What that post describes is simply a bad approach to ab test design. Measuring impact is orthogonal to creativity and good strategy. It's not the test's fault if your design is uninspired or your strategy piecemeal, and likewise declining to accurately measure the impact of an 'authentic' and outside-the-box type feature doesn't mean your design is actually achieving its goals--you've only found an excuse to put your head in the sand and avoid accountability.
I am not sure what kind of service this site provides, but it is region locked!
When I try to access the site, (any part of the URL, even the homepage, I get to see the following error:
"Sorry, this request is coming from a country which we currently do not support. If you feel this is in error, please contact our support team by emailing support@thumbtack.com."
I guess even blogs and articles are now coming under region protection.
That is pretty funny, I wrote the code for that blacklist like 5-6 years ago when Thumbtack was being hammered with spam requests from China and India. I thought it only blocked POST requests, though maybe my memory is wrong or they have made the block more expansive in the meantime.
Anyways there is a version or ABBA usable on github or you can always get the source and run it locally:
It's probably a cloudflare instance that they're running on top of the page. CloudFlare's default protection scheme is pretty harsh to non US/European users.
I've gotten blocked from a couple of sites I was managing because I was in Japan.
1) The multiple testing correction, which many people ignore when testing > 1 variants vs. the control
2) They display not only the p-value, but the confidence interval around the _difference_ between the variants (i.e. the effect size). It's tempting to observe an increase of say 50% that is statistically significant and think that you can be confident about the 50%. This is not the case! You chose your sample size to be confident that the actual effect size is > 0 -- not that you would have a tight confidence interval around whatever difference you observed. Depending on your sample, the lower bound could still be close to 0.