On the positive side, it's unlikely "everyone" who isn't a government agency with billions of dollars of taxpayer money to waste has the time, storage, and compute resources to mount the kind of attack you're talking about.
I know NSA is the topic of the day, but they're kind of a special case here. They're the single most powerful and well funded adversary the average crypto user will face.
The difference is when your ex releases your key and you are using this system then everybody can read your email instead of just the NSA and the recipient.
So, what you are saying is that nobody else, but you, should ever have access to your private key, right? I'm pretty sure that's PKI 101, which I think was Karunamon's point; and, I'm pretty sure the solution to your proposed 'weakness' in this system is not technical.
I know NSA is the topic of the day, but they're kind of a special case here. They're the single most powerful and well funded adversary the average crypto user will face.