Hacker News new | past | comments | ask | show | jobs | submit login

For your first question: https://platform.openai.com/tokenizer



I saw that, but the language makes me think it's not quite the same as what's really being used?

"how a piece of text might be tokenized by a language model"

"It's important to note that the exact tokenization process varies between models."


That's why they have buttons to choose which model's tokenizer to use.


Yes, thank you, I understand that part.

It's the might condition in the description that makes me think the results might not be the exact same as what's used in the live models.


The results are the same.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: