What Are AI Tokens? How Chatbots Split and Count Text

A token is a small piece of text that an AI language model treats as one unit. It may be a whole common word, part of a longer word, a single character or a bit of punctuation. Models work with numbers, so the first step with any text is to cut it into tokens and give each one a number. Limits on how much a chatbot can read, and in many cases the bill for using it, are counted in tokens rather than in words.

Why text is turned into numbers

A language model is a mathematical system, and it cannot work on letters directly. Wikipedia's article on large language models puts it simply: because the algorithms process numbers rather than text, the text has to be converted to numbers. The conversion is called tokenization. A fixed list of pieces, called the vocabulary, is chosen. Each piece in the list gets its own whole-number index, and the model learns what each index tends to mean.

Why not just use words, or just letters?

Both simple options have problems, as the Hugging Face documentation on tokenizers explains. If every word is a token, the vocabulary has to hold every form of every word: "love", "loving", "loved" and "lovingly" each need their own entry. The list becomes enormous, which makes the model bigger and slower, and any word missing from the list cannot be handled at all.

If every character is a token, the vocabulary is tiny and nothing is ever missing. But sentences become very long strings of tokens, and a single letter such as "l" carries far less meaning than the word "love". The documentation says this hurts performance.

The middle route is called subword tokenization. Common words stay whole as one token, and rare or new words are broken into smaller pieces that are already in the vocabulary. The Hugging Face page gives "annoyingly" as an example, which might be split into "annoying" and "ly", or into "annoy", "ing" and "ly", depending on the vocabulary. Wikipedia shows that an older GPT-3 tokenizer split "tokenizer" into "token" and "izer", and "tokens" into "t", "ok" and "ens". So tokens often do not line up with words.

How the pieces are chosen

The most widely used method is byte pair encoding, or BPE. The Hugging Face documentation describes it in steps:

• Start with every single character as its own unit.
• Find the pair of neighbouring units that appears together most often in the training text.
• Merge that pair into one new unit and add it to the vocabulary.
• Repeat until the vocabulary reaches the size that was chosen in advance.

The documentation works through a tiny example. In a text with the words "hug", "pug" and "hugs", the letters "u" and "g" appear together most, so "ug" becomes a unit. Frequent combinations therefore become single tokens, and unusual ones stay as smaller parts.

Vocabulary sizes are chosen by the model's designers. The Hugging Face page says the original GPT model used BPE with a vocabulary of 40,478 entries, and GPT-2 used 50,257. GPT-2 used a version that works on bytes, the basic units of computer storage, so any text at all can be represented without an "unknown" token. The page also names other methods, called Unigram and WordPiece, and a library called SentencePiece. SentencePiece treats the space as an ordinary character, which helps with languages such as Chinese and Japanese that do not put spaces between words.

Why the same sentence can cost different amounts

Every model family has its own tokenizer, with its own vocabulary, so the same text can produce a different number of tokens in different systems. Anthropic's documentation on token counting makes this concrete. It states that Claude models from version 4.7 onward use a newer tokenizer, and that the same input text produces approximately 30 percent more tokens than on earlier models. The exact increase depends on the content. Anthropic therefore advises recounting prompts against the model you plan to use instead of reusing counts from an older one.

Text that is common in the training data tends to need fewer tokens, because its pieces were merged into the vocabulary. Rare words, unusual names and text in less common languages tend to be split into more pieces. The sources above do not give a single words-per-token ratio, and it varies by model and language, so any such figure should be treated as a rough guide.

Where token counts show up

Anthropic's documentation says that counting tokens before sending a message helps you manage rate limits and costs, choose between models and fit a prompt to a target length. It describes a counting endpoint that returns the number of input tokens, and notes the result is an estimate that can differ slightly from the number actually used. It also says billing reflects only your own content, not tokens the company adds for system optimisation.

This also explains the "context window" of a chatbot. That window is a limit on how many tokens, from your messages and the replies together, the model can take into account at once. A page that looks short in words may use more of that limit than expected if it contains code, numbers or unusual names.

The short version

Tokens are the pieces AI models actually read: chunks of text turned into numbers using a fixed vocabulary built from patterns in training text. Words, tokens and characters are different measures, and the exact count depends on the tokenizer, which differs from one model to the next.

Sources

• Token counting, Claude API documentation, Anthropic
• Summary of the tokenizers, Hugging Face Transformers documentation
• Large language model, Wikipedia

Image credits: Cover photo: Flickr in Scrabble letters by Taro Taylor from Sydney, Australia, CC BY 2.0. Pictures from Wikimedia Commons.