Did you try entering one of those scrambled sentences into the tokenizer? It's n... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		nverno on Dec 9, 2022 \| parent \| context \| favorite \| on: ChatGPT, Rot13, and Daniel Kahneman Did you try entering one of those scrambled sentences into the tokenizer? It's not tokenizing words exactly- the doc says roughly ~100 tokens => 75 words.

pa7x1 on Dec 9, 2022 [–]

I just tried out of curiosity. The jumbled words get broken up into different tokens, the unjumbled words do not get split.

https://imgur.com/a/a3zmkIv

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact