That's the meme, and it probably did go that way (crazy -> eccentric), but it's also true that someone who was just a little bit off would have been called eccentric whatever their social class.

It would not have been a very effective euphemism for a crazy rich person if it didn't also have the nicer meaning for everyone, rich or poor.