I am interested in how a 32k token context even works. That is so much larger than 4k that I am having a hard time imagining how prompts will change and what sort of output is now possible. That is 50 pages of text. Far larger than most content currently being consumed and generated by LLMs.
Q&A and summarization it will be easy to see improvements as current recursive summarizing and embedding techniques are very "lossy" but outside of improving current use cases what will now be possible??
This is a game-changer, because now companies will probably be able to provide the _complete_ context regarding a specific business problem / use case, and have GPT either solve their problem or create useful output.
For example, let's say I have an issue on GitHub that describes some implementation task. With a 50-page context size, we could probably provide to that context the entire source repo, 5-10 relevant issues, and then the issue in question, and GPT will be probably be able to complete it end-to-end
Yea this is huge. Been playing with conversational technology in langchain and one of the issues you have to manage is the historical conversations, langchain has some cool ways to deal with it but this changes the nature of the problem entirely.
Q&A and summarization it will be easy to see improvements as current recursive summarizing and embedding techniques are very "lossy" but outside of improving current use cases what will now be possible??