The Markov chain wouldn't fit in a googol of observable universes. The analogy is absolutely misleading. There's no reason to think that the chain of that humongous size cannot capture at least something of what humans with their limited brain size can do.

It cannot capture non in-context learning, true. But we have tools to modify the "Markov chain".