Your logic is flawed in that shares aren't simply "destroyed" they have to be bought back on the open market at an agreed upon price from a willing seller. The sellers of the shares may not want to sell and will therefore require more than the perfect price 𐤃X that accounts for adjustment of market cap based on reduction in shares.

In theory if no one wants to sell shares of Apple during a buyback the share price will head towards infinity. There's always a price though that someone will let go of a share at.