Because passport is just a piece of paper (or a record in a database) that armed dudes called "border control policemen" are asking for to let you go. And if you don't have one, they'll point a gun on you.

Make a thought experiment what would happen if you try to peacefully move to another country without a passport.