tokenizer

AI·1 November 2026·5 min read

What Are AI Tokens? How Chatbots Split and Count Text

AI chatbots do not read letters or whole words. They read tokens, small pieces of text. Here is how text is split into tokens, and why the count matters for limits and cost.