To be fair, cryptography and number theory have coexisted without much overlap from centuries (millennia?). Then, the rise of mechanized cryptanalysis forced us to look for hard-to-break ways to encrypt stuff, and prime factorization was a very good candidate.