This is neat, in a look-what-i-can-do sort of way. And I do appreciate the write-up. However.... (and there's always a but) does this page convert better? And if it doesn't, will they have the guts to rollback?
I don't know anything about the politics behind github and the employees. I'm sure they're all wonderful people. This is just a projection from my company. Even if the new flashy design performed 20% worse, it wouldn't be gotten rid of. They would say something like "well, 20% isn't that bad. it just needs time to bed in" or something like that. Fast-forward a year, and we still have the crappy performing good-looking page!
I think this question is just as much about data as it is intuitive and feelings. Does a rollback make the front page of HN? Does the newer version leave a lasting impression that can't be measured? I don't think you can truly measure these things without actually measuring sentiment, etc.
I'm sure Netflix has seen increased metrics across the board but everything they've done and added in the last three years or so has turned me against the brand. As a very happy subscriber since 2002 I stopped in 2019. Anecdotal and maybe just me but I don't think everything is about some 28-day conversion metric.
I think it's more about the guts to do something you believe in and trusting the people you hire than just metrics.
Yeah. You can optimize as many metrics as you want, but you never have all the metrics. You can optimize your kid's soccer/chess/piano/judo skills too. Maybe it will allow them to live the life of their dreams. Or maybe you completely crush their happiness. I know it's not the same, but hopefully people will get the idea. If something doesn't look healthy, trust your instinct too and not only the metrics. They don't capture everything.
I don't know anything about the politics behind github and the employees. I'm sure they're all wonderful people. This is just a projection from my company. Even if the new flashy design performed 20% worse, it wouldn't be gotten rid of. They would say something like "well, 20% isn't that bad. it just needs time to bed in" or something like that. Fast-forward a year, and we still have the crappy performing good-looking page!