But if you're measuring via proxies you lose causation. Have those gradual changes that immediately increased conversion or upsell by 2% over the last 3 months cheapened the brand and site, causing long-term damage to its perception in the market place? Has the move to weekly newsletters, that brought that burst of business started to make people think of you as irritating and spammy? A/B testing isn't going to tell you that but those blessed metrics are going to start going south down the road and you aren't going to be able to A/B test your way out of it. Perceptions last, brands are ephemeral and empirical, careful testing indicated New Coke should have been a massive hit.