“Struggle” at what? Struggle to have enough data to get smarter? Struggle to perform RAG and find legitimate sources?
I don’t think that we are going to get big improvements in LLMs without architecture improvements that need less data, and the current generation of models appears to be good enough at creating content from data/knowledge to train any future architectures we have with better synthetic datasets. Fortunately we have already seen examples of both of these “in the lab” and will probably see commercially sized models using some of the techniques in the coming months.
My point is about the inevitable future when _those_ models start to struggle.
The phi approach doesn’t seem like breaking the ouroboros, it just feels like inserting another model/snake into the loop.