I am like you. Every big performance win i ever got was from changing algorithms. Sometimes i didn't write either the old or the new, but just swapped out a different implementation. A good handle on that stuff can get you a long way. And can certainly be picked up from reading. it's a lot closer to just math.

That said, spend a week with C and valgrind/cachegrind. There's a lot of theoretical stuff that is hard to get at (or is for me) without a little exposure to how the system works. a coupe hours here and there will extend your mental model to include the various layers of cache. That'll make the more esoteric stuff more accessible.