I've never noticed the guards changing when going through town. What I /have/ noticed is the swell of tourists that surrounds the gates every time it's happening and how it always seems to be when I'm trying to pass through, but maybe that's too personal a grievance...