It doesn’t seem to work quite like that. I don’t know the technical details, but I definitely see replies from people I don’t follow, from other instances, on posts of people I don’t follow on different (again) instances from mine and the replies.
Perhaps it could be that I’m seeing replies from people that others on my instance follow? Or perhaps there’s some other mechanism to fetch replies.
Probably that. Actually if your server receives the reply for any reason, it displays it, usually. There isn't a mechanism to fetch replies, so this is a kind of accidental filtering, while ATproto is the opposite and tries to make everything globally visible.
I use my own server with only me, so it only receives replies because I'm following the person who replied or their whole-server feed.
Perhaps it could be that I’m seeing replies from people that others on my instance follow? Or perhaps there’s some other mechanism to fetch replies.