Token
A token is a unit that a tokenizer turns into an identifier for a model to process. In text models it may represent a word, part of a word, punctuation, or another text fragment.
[Hugging Face]In practice
A team measures a long support message with the model’s tokenizer before sending it, rather than assuming one word equals one token.
[Hugging Face]A little deeper
Token boundaries depend on the tokenizer. Counting words is therefore different from counting tokens; the same text can produce different token sequences with different tokenizers. [Hugging Face]
A common mix-up
One token always equals one word.
A word may be split into several tokens. [Hugging Face]