Right, the issue is that nobody checks it and they don't really tell users upfront why they need to. It also notifies you if the number changes later, but seems like that can happen for benign reasons too, so it might go ignored. WhatsApp and Telegram are similar.

Someone could orient an E2EE app around these trusted identities, but it'd probably not be very popular.