Correctly price in the negative externalities caused by burning petroleum and people can go have their fun in ICE cars away from major population centers. It'll be more expensive, for sure, but I don't think it'll only be in the realm of the rich. Gas is already something like $8/gallon in many parts of Europe and is still widely used, just for daily life (forget racing).
Why would that be? Insurance is about covering accidents; why would accidents suddenly shoot up for ICE cars? I haven't seen any data suggesting that ICEs are inherently more likely to get in accidents than EVs.
Now if you want to talk about advanced driving aids, all the way through to self-driving, then sure, not having those will result in higher insurance. But that's not tied to the engine, and there's plenty of non-EVs that have very advanced driving aids.