Save a browser cookie when the login is initiated. When the link is clicked check if the same cookie is present. If not, ignore it. Expire the link and the cookie after n minutes.
Surely this breaks the "email is not on same device as login" use case? At least with normal magic links, they're merely incredibly annoying but doable (via e.g. typing in the URL)
That use case still works. In fact it works better because if you click the link on your phone you don't automatically get logged in on your phone browser (or your email client's in-app browser). You can then copy the same link on your desktop and it will work as expected.