Explains tokens and tokenization, how text maps to tokens across tokenizers, impacts on model context and cost, and tools and best practices for accurate token counting.
When you send text to a model it doesn’t read words the same way people do — it processes tokens. This lesson makes that concrete by using a tokenizer and observing how different inputs map to tokens. Seeing how text becomes tokens helps you reason about two crucial things: how much an interaction costs and how much information a model can process at once.
Tokens are the atomic units models process. Tokenization can split words into subword pieces, include punctuation and whitespace, and vary by model family. Use a tokenizer when you need precise token counts for budgeting or capacity planning.
The tokenizer used in these examples is tiktokenizer.com. It’s straightforward: paste or type text into the prompt area on the left, pick a tokenizer at the top, and the tool shows how the text is split into tokens and the numeric token IDs on the right. Everything updates live — no submission required. Try different tokenizers because the same text often tokenizes differently across model families.Let’s start with a short English sentence:
The quick brown fox jumps.The·quick·brown·fox·jumps.976, 4853, 19705, 68347, 65613, 13Token count6
A human sees five words; the tokenizer shows six tokens — the extra one is the period. Each colored piece in the tokenizer view is a token and the number underneath is its token ID. Remember: the model processes token IDs, not the raw characters.A token is not the same as a word. Sometimes a whole word maps to a single token; sometimes a word is split into multiple subword tokens depending on the tokenizer vocabulary and rules. Formatting, punctuation, and whitespace matter. For short sentences that difference is small, but for long prompts, documents, or source code those differences add up because the model handles everything you send.Now let’s try a very long, uncommon word. I looked one up on Merriam‑Webster and pasted it into the tokenizer:
Because uncommon words may not appear as single entries in a tokenizer’s vocabulary, the tokenizer often breaks them into multiple subword pieces. Example:
Numbers and non‑Latin scripts often split into multiple tokens depending on the tokenizer’s vocabulary and segmentation rules. For example, the digit string “1234567890” can be broken into several tokens:
1234 5678907633, 19354, 29338, 15Token count4
Emojis and other special characters may also expand into multiple tokens. For instance, three emojis mapped to six tokens with one tokenizer:
If you switch the tokenizer (for example, from a GPT-5.6-style tokenizer to a GPT-2-style tokenizer), the same emoji sequence can produce a different tokenization and token count:
Tools like tiktokenizer may indicate whether a tokenization is exact or estimated. Some models publish their encodings so tools can reproduce exact counts; others require estimation. Always check whether the tokenizer you used provides an exact or estimated count before relying on it for budgeting or capacity planning.Chat-style requests typically include extra structure beyond a user’s visible text — system instructions, assistant messages, conversation history, retrieved documents, and tool definitions. Those all contribute to the overall token count. Even a short chat exchange including system delimiters can add many tokens:
`<|im_start|>system<|im_sep|>You are a helpful assistant<|im_end|><|im_start|>user<|im_sep|>Evaluate this markdown file and provide a summary for a devops qa engineer<|im_end|><|im_start|>assistant<|im_sep|>`200264, 17360, 200266, 3575, 553, 261, 10297, 29186, 200265, 200264, 1428, 200266, 112737, 495, 116713, 1974, 326, 3587, 261, 18522, 395, 261, 3947, 4645, 25002, 36894, 200265, 200264, 173781, 200266Token count30
Because system prompts and chat-format delimiters consume tokens, consider the entire request payload (system messages, conversation context, retrieved data) when estimating usage — not just the user’s current message.Tokens also translate directly to cost: models are typically priced by the number of tokens processed (input + output). For example, a model’s pricing might list a “price per 1M tokens” for input and output — see OpenAI’s pricing for reference. A tiny 30-token prompt costs almost nothing, but as you add context, documents, or request longer outputs, costs increase rapidly.
When choosing a model, balance capability against cost. Frontier models deliver strong capabilities but are expensive per 1M tokens; smaller or specialized models are usually more cost‑effective for many tasks.Tokenization examples at a glance:
Track the entire request: include system prompts, conversation history, retrieved docs, and tool definitions when estimating tokens.
Prefer token-aware tools or libraries (like official encodings when available) for exact counts.
Remove unneeded history, compress or summarize long retrieved documents, and use smaller models for tasks where high capability isn’t required to reduce cost.
Monitor token usage and cost regularly; token behavior can change across model versions and tokenizer updates.
Estimating tokens with the wrong tokenizer may mislead budgeting and capacity planning. Always confirm whether a tokenizer reports exact or estimated counts and use the encoder associated with the target model when accurate accounting matters.
Summary
Tokens are the unit models process; text is converted to token IDs before the model handles it.
Tokenization depends on the tokenizer and model family; the same text can yield different token counts across models.
Token count affects how much information a model can consider (the context window) and how much an interaction costs.
Always consider the full request payload — system messages, history, retrieved data — when estimating token usage, not just the visible user input.
