Gradual polar shift happens over millions of years; specifically characteristic rate of true polar wander is 1° or less per million years.
If the shifting occurs as a “Cataclysmic” polar shift [1] or geomagnetic reversal [2], the periodic reversal of the Earth's magnetic field (effectively switching the north and south magnetic poles) — I agree that it would likely have terrifying results for the inhabitants of Earth.