This is definitely a per-person sort of thing. I was a maths major who took several physics courses and I tended to find that they muddied my mathematical understanding as often as they helped whereas I found algebra and number theory (my foci, admittedly) to be easy enough to understand on their own that the classes were just on this side of trivial.

Physicists often take shortcuts based on physical intuition, e.g. symmetry arguments.

I remember having trouble understanding equilibrium charge distributions when studying E&M -- laws such as "at electrostatic equilibrium, all charge is on the surface of the conductor" -- the questions of whether such a static equilibrium exists, is unique, or is a place you'll always end up from an arbitrary initial configuration weren't really addressed. (The best I could come up with is that any movement of charge will eventually die out due to friction, but this was more of a vague intuition than a satisfying explanation.)

Anyway, I guess if you have the physics gene, you just have a strong intuition that tells you the answers to questions like these. I didn't have it.

Physics is better for people who like to trust their intuition. Math is more programming-like in that the people who do well tend to be hard-nosed about details and corner cases.