The kernel used by Android is completly irrelevant to userspace and can be changed tomorrow at Google's will, and only OEMs and rooted devices will actually notice it.
On embedded there are plenty of OSes to choose from, with FOSS embedded industry support moving to MIT/BSD licensed kernels, as OEMs don't want to deal with shipping GPL code on embedded devices, specially after v3 happened.
The fundamental reason is that there is no big company pushing Linux on the Desktop. The Linux Desktop community has thus far proven, if I'm being frank, completely retarded at making anything people actually want to use.
It's not as bad as you say. Ubuntu has made a decent attempt, with some (albeit minor) success. But Ubuntu seems to be moving away from desktop to also focus on cloud stuff. Valve has done a lot of work with Windows compatibility.
One thing I'm observing is that macOS and Windows are basically regressing to have the same type of nonsense and quality issues that are indeed typical on desktop Linux. It's possible (but not certain) that they'll eventually cross paths. Linux might not take over the consumer desktop market (unless it's under a proprietary shell, like Android / ChromeOS), but Linux could possibly take over the high-end/power user/workstation market.
Maybe, but that will still either require some big player to come in and make sane decisions to tear the Desktop portions away from the garbage way they're done now, or for the community to get it's act together. I have been waiting 20 years for the latter and do not expect it to ever happen at this point.
There is no fundamental reason why it can't eventually take over gaming, or even desktop.