Indeed. Unfortunately, that's fundamentally the way public-key crypto works. If you do try to send a message to an email with no public key, it will prompt you to send an invite instead.