I started using Visual Studio Code exclusively around 2020 for C# work and it's been great. Lightweight and fast. I did try Rider and 100% it is better if you are open to paying for a license and if you need more powerful refactoring, but I find VSC to be perfectly usable and I prefer its "lighter" feel.
I still think Visual Studio is better, but you can easily work on small to mid-size projects in VSCode. Could you use Vim? I probably wouldn't, but you can say the same for Java.
How's the LSP support nowadays? I remember reading a lot of complaints about how badly done the LSP is compared to Visual Studio.