It's a catch-22 because if you stop innovating your UI for 20 years and alternatives come up with something people actually like then you will lose users to them and slowly fade into irrelevancy.
Firefox succeeded because it was a fresh take on the entire browser UX at a time when Internet Explorer had been stagnant for half a decade.
As I remember it, Firefox succeeded because it fundamentally worked well and was very configurable, not because of the UX. The others at the time were bad at both of those things.
The UX of Firefox was (and, I'd argue, still is) not great, but it made up for that by being configurable enough that you could fix it for yourself.
Firefox (Phoenix / Firebird) initially got its userbase from the Mozilla userbase which was this capable browser with many features and flexibility (extensions), but it was bloated, slow, with a kinda outdated (non-native) UI. Firefox's differing feature was its lightness and (native-like, per-platform) UI freshness, not configurability.
FF success against IE was mainly caused by Microsoft completely dropping the ball by simply not developing IE for a couple of years. I think one of the elementary features which has driven mass FF adoption was tabbing support, which IE6 didn't have.
Everyone I knew who wasn't working in tech that switched to Firefox did so because tabs were this revolutionary new thing that transformed their browsing experience.
Of course, Firefox wasn't the first browser to offer tabs, but it was the first that was also fast, highly compatible, and easy for the average person to pick up and use. It's predecessor, the Mozilla suite, included all the same extensibility and customization, but it was also a bloated mess that nobody wanted to use.
In my opinion, Firefox has never been particularly good. I tried it when it first came out - it was slow and a memory hog. Using IE6 was a much better experience, so I stayed there. It was only once Chrome came out that we had an actual good alternative to IE, and I switched immediately once I tried it.
I would love for Firefox to be awesome, but ultimately (in my subjective opinion of course) it just kind of sucks and always has.
Firefox succeeded because it was a fresh take on the entire browser UX at a time when Internet Explorer had been stagnant for half a decade.