Part of the writeup included my app's core interaction, which is a six stage funnel which, at the time, had 48.5% conversion. Mixpanel showed me which two stages were screwing things up. I fixed them.
The funnel conversion is now 56.7%, where is a 16.9% lift in conversions. (Statistically significant at 90% confidence level, but it wasn't A/B tested.) For the detailed breakdown, see the submission link.
Thought you should know, we added the ability to see Google keyword performance throughout every stage of the funnel. This happens automatically with no extra work.
This is generally for those of you who want to see how certain keywords (e.g. Adwords) effect the overall conversion rate and effect your bottom line.
Awesome Suhail. I think I should point out that you would have to do extra work if you were using my Rails Mixpanel integration, since it never loads your Javascript and thus you are never given access to browserland things like the referrer string.
But that is a good automatic feature to most of your customers, who use the JS integration rather than a server-side API library.
Thanks for this great article and thanks for sharing the code! Methinks I'll be installing it sometime in the next week.
Quick questions about your A/B testing...
* What kind of user volumes are you looking to do A/B testing on? How does that relate to how long each of your tests runs/will run?
* Are you running several A/B tests in parallel?
* Do you make each user an A or B, or do you keep the A/B testing page-based? I'm assuming the former, since otherwise the same user might see both sides of the test, which is not desirable.
Note: I did not A/B test the experiment covered in the Mixpanel article. I did just release an A/B testing framework (A/Bingo) today. You can find it here:
1) I run tests until I get statistically significant results. How long that takes is outside the scope of this comment, except to say that if you're testing something with a high conversion rate and testing two very different alternatives you can get significant results in ~250 users or so. This lets me do very rapid iterating. Changes in my core interaction can be A/B tested in under a week, which works well for me (I do most development on weekends).
2) I run lots of tests in parallel. For why that is not as bad an idea as you might think it is, see the A/Bingo FAQ.
3) That is a core design consideration for A/Bingo. I obsessively track users to make sure they always get the same alternative. A/Bingo is the only Rails A/B testing framework that does this, to my knowledge.
http://news.ycombinator.com/item?id=749497
Part of the writeup included my app's core interaction, which is a six stage funnel which, at the time, had 48.5% conversion. Mixpanel showed me which two stages were screwing things up. I fixed them.
The funnel conversion is now 56.7%, where is a 16.9% lift in conversions. (Statistically significant at 90% confidence level, but it wasn't A/B tested.) For the detailed breakdown, see the submission link.