Token
A token is a unit that a tokenizer turns into an identifier for a model to process. In text models it may represent a word, part of a word, punctuation, or another text fragment.
[Hugging Face]In practice · hypothetical example
A team measures a long support message with the model’s tokenizer before sending it, rather than assuming one word equals one token.
[Hugging Face]A little deeper
Token boundaries depend on the tokenizer. Counting words is therefore different from counting tokens; the same text can produce different token sequences with different tokenizers. [Hugging Face]
A common mix-up
One token always equals one word.
A word may be split into several tokens. [Hugging Face]
Can a single word become several tokens?
Sources & editorial notes
Evidence: supported. Primary-source support for this scoped entry; publication approved by the project owner.
- Tokenization algorithms ↗ (opens in new tab)Hugging Face · Publication date unknown
Relevant section: Subword tokenization
Last editorial review: 2026-09-13 by project-owner.
First observed in this corpus: Unknown.
Revision history
Revision 2 · Created 2026-09-13 · Updated 2026-09-13
Project owner approved the current content for publication. Existing evidence scope and limitations remain applicable.