Anyone who works at your operator can pull this off, or anyone that can bribe one of the former.

Then there are other ways to achieve the same thing without SIM swaps. SS7 level rerouting can be done globally ("this phone is now roaming from within our network").