Replaced Desktop with `colima` as well few months ago. I've been using it daily since then. I did not have any issue, sometimes I just delete / start a instance to upgrade the docker version, it only takes few minutes.

I like the fact that I decide when I upgrade, not Docker Desktop nagging me every week.