As a dev who works with telemetry in my day job I HATE when "better metrics" gets in the way of a feature or product improving.
(I also hate the pervasiveness of identifiable tracking but that's a whole different rant.)
You don't need to perfectly A:B test your two different offerings if that means preventing users from using the product in a way that best suits them. Stop putting shit behind "read more" buttons just because you want to see if people are actually reading your articles. Stop breaking things into multiple screens just so you can track your damn funnels. Better MAU _tracking_ (not even better MAU) does nothing for user experience and pursuing that at the expense of everything else is just plain foolish.
Yeah, it becomes just about getting a "better metric" which is in fact a number that sounds better to others instead of using metrics to improve user experience.
(I also hate the pervasiveness of identifiable tracking but that's a whole different rant.)
You don't need to perfectly A:B test your two different offerings if that means preventing users from using the product in a way that best suits them. Stop putting shit behind "read more" buttons just because you want to see if people are actually reading your articles. Stop breaking things into multiple screens just so you can track your damn funnels. Better MAU _tracking_ (not even better MAU) does nothing for user experience and pursuing that at the expense of everything else is just plain foolish.