If its context size is >= 25k words, yes. Otherwise it will just discard the start of the prompt. And it’s a sliding window, so the more it generates, the more it forgets.
You could get an 'Illuminatus!' type book out of this, especially if you steered the ending a bit in order to reference earlier stuff. If you're trying to make a sprawling epic that flings a kaleidoscope of ideas, GPT can do that sort of thing, it's just that it won't end up making sense.
GPT is going to be rather poor at priming people for an amazing ending by seeding the ideas and building them into the narrative. Though if you're directing it with enough granularity, you could tell it to do that just like you'd tell yourself to do that when you're doing the writing yourself.
But then you're becoming the executive writer. On a granular enough level, the most ultimate executive control of GPT would be picking individual words, just like you were writing them yourself. Once you want to step away and tell it to do the writing for you, you drift more into the GPT-nature to the point that it becomes obvious.
