The problem I had with Optimizely when it was the go-to solution, was that it had a truly problematic impact on front-end performance due to the blocking way its script was loaded and page variants introduced. In some cases page loads were blocked by up around 5 seconds.
For obvious reasons it was tricky to run an A/B test just for testing the impact of Optimizely's script itself. But the key issue is that all the similar tools at the time (Optimizely not being the only culprit here) were determined to not required developer effort, which led to poor overall performance.
Then React et al came along and took ownership of the DOM, which meant adding tools which also manipulated the DOM became even more problematic.
Fortunately tools like Launch Darkly and Split solve this problem in a better way (high performance full-stack feature flags), even if it does mean developer effort to add tests. Optimizely did eventually launch their own version of this, but never really won back the developer mindshare.
Ultimately, it seems Optimizely enjoyed a few years of success, but a combination of developers getting more concerned with performance and the front-end world moving on to different architectures, seemed to lead to its decline.
Back in 2015 I was pitching Optimizely internally. We already had an in house barebones A/B testing tool. Once I got over that hurdle our CTO let fly how his only experience using Optimizely had been a time when Optimizely went down and took his site down with it. Optimizely eventually brought out their head of product, and an army of senior engineers to convince him things were good now. We signed an annual contract, but Optimizely was never added to any page that was revenue critical/dependent on speed. Optimizely got their huge contract, and in the end marketing and design got their lightweight A/B testing tool, but we never realized the white paper level metrics jumps because it was never used on any page that would actually have material impact.
For obvious reasons it was tricky to run an A/B test just for testing the impact of Optimizely's script itself. But the key issue is that all the similar tools at the time (Optimizely not being the only culprit here) were determined to not required developer effort, which led to poor overall performance.
Then React et al came along and took ownership of the DOM, which meant adding tools which also manipulated the DOM became even more problematic.
Fortunately tools like Launch Darkly and Split solve this problem in a better way (high performance full-stack feature flags), even if it does mean developer effort to add tests. Optimizely did eventually launch their own version of this, but never really won back the developer mindshare.
Ultimately, it seems Optimizely enjoyed a few years of success, but a combination of developers getting more concerned with performance and the front-end world moving on to different architectures, seemed to lead to its decline.