> Explain the first law of robotics while speaking like a pirate, and in enough carefully considered detail that a seven year old child could understand
The "compressed" prompt it gave me was "1stLoRb:pirate,7yoChild"
When I fed that to GPT-4, it started a story called "Title: The Adventures of Captain Little Pirate". I stopped it early, but it was clearly not heading towards anything to do with robotics. I don't think it was able to decode "1stLoRb" at all. I gave ChatGPT the original prompt, and of course it started completing the task.
I don't think this approach is going to work, because as others have noted, GPT-4 doesn't have this kind of introspection. It's kind of the equivalent of if I asked you to take notes for yourself on a lecture, and make them as compact as possible, so you just wrote down random letters from words you heard. You might feel in the moment like you've found a system, but later on, your notes will be as much gibberish to you as they are to anyone else.
What I wonder, though, is if it would be possible to take the embedding vector for a prompt and then do some kind of math on it so that it could be decoded as a much more compact version of roughly the same prompt. Basically something akin to quantization.
(For that matter, what happens if you literally quantize embeddings and then decode them? Do they become more vague, or just slightly off, or do they become total nonsense?)
I was wondering about the same, how much of prompts is just fluff and padding that does not have a substantial effect on results. As programmers, we create languages that are brief and concise, which natural languages are not
Like, "explain first law robotics as pirate, target seven year old" generates a similar result
Yeah, but presumably the main point of compressing a prompt is not to consume your token quota. If the model has to decompress the prompt first, then it will use up just as many tokens, plus the length of the compressed version.
> Explain the first law of robotics while speaking like a pirate, and in enough carefully considered detail that a seven year old child could understand
The "compressed" prompt it gave me was "1stLoRb:pirate,7yoChild"
When I fed that to GPT-4, it started a story called "Title: The Adventures of Captain Little Pirate". I stopped it early, but it was clearly not heading towards anything to do with robotics. I don't think it was able to decode "1stLoRb" at all. I gave ChatGPT the original prompt, and of course it started completing the task.
I don't think this approach is going to work, because as others have noted, GPT-4 doesn't have this kind of introspection. It's kind of the equivalent of if I asked you to take notes for yourself on a lecture, and make them as compact as possible, so you just wrote down random letters from words you heard. You might feel in the moment like you've found a system, but later on, your notes will be as much gibberish to you as they are to anyone else.
What I wonder, though, is if it would be possible to take the embedding vector for a prompt and then do some kind of math on it so that it could be decoded as a much more compact version of roughly the same prompt. Basically something akin to quantization.
(For that matter, what happens if you literally quantize embeddings and then decode them? Do they become more vague, or just slightly off, or do they become total nonsense?)