Refs are just as immutable in that perspective. And no, you're not supposed to mutate ref.current.anything outside useEffect or a callback or similar...
Of course you can! That's the whole point of refs, and why you can't use them as a dependency to callbacks or effects. They're totally mutable, which makes them very tricky to deal with. I mean, I wouldn't recommend updating them outside of effects/callbacks, but it's valid. For example: you can make a ref to count the number of times React calls your render loop by updating it in the body.