Definition
Token (in LLMs)
A token is the smallest unit of text a language model processes, usually a word, part of a word or a punctuation mark, into which text is split before processing.
As a rule of thumb, one token in English roughly equals four characters or about three-quarters of a word; German often uses more tokens because of longer words. Tokens matter because many models' context length and pricing are measured in tokens.