Tokens sit between human language and the mathematics of modern AI. They determine how text is represented, how much information a model can handle and, how much using that model costs. Yet a token is neither a word nor a thought.
Ask an AI company how much its model can read and the answer will probably be given in tokens. Look at an API price sheet and you will be charged by the million tokens. Inspect a developer dashboard and you may encounter input tokens, output tokens, cached tokens and reasoning tokens.
The word has become one of the basic currencies of generative AI. But a token is not a word, a thought or a fixed quantity of information. It is better understood as part of the translation layer between human language and the numerical machinery of a language model.
A token is not a word
Humans see a sentence as words separated by spaces and punctuation. A language model usually does not.
Before most modern large language models process text, a tokenizer divides it into units drawn from a fixed vocabulary. One token might correspond to an entire common word. Another might contain part of a word, a punctuation mark, several letters or even whitespace.
OpenAI gives a useful rule of thumb for English. One token averages roughly four characters, or about three quarters of a word. The precise result depends on the tokenizer, the language and the text itself. A long familiar word may require fewer tokens than a shorter unusual one. A common suffix may have its own token, while an uncommon name might be split into several pieces.
So there is no universal conversion such as “1,000 words equals exactly 1,300 tokens.” There cannot be, because tokenization is partly determined by what the tokenizer learned was useful to keep together.
Why not simply give the model whole words?
New names appear constantly. So do technical expressions, spelling mistakes,code, slang and words borrowed between languages. Many languages also contain extremely productive systems of prefixes, suffixes and compound words. In practice, that would mean building a complete linguistic bible covering conjugations, inflections, proper names, or every new expression humans will continue to invent in the future. It would be an almost impossible task and, more importantly, a never-ending one.
The opposite approach would be to eliminate the problem altogether by processing text character by character. But in that case, a single sentence would be broken down into a much longer sequence, making it more computationally expensive to process. Subwords offer a useful compromise. Very common sequences can be treated as a single unit, while rarer ones can be reconstructed from smaller pieces. This approach is known as Byte-Pair Encoding, or BPE. Research that helped popularise its use in neural machine translation showed that it could handle rare or previously unseen words by breaking them down into smaller units.
Google’s SentencePiece tokenizer later introduced a method capable of learning these segmentations directly from raw text, without relying on the word boundaries specific to any particular language. Tokenisation may seem like a minor preprocessing step, but it ultimately determines the basic building blocks that the model receives.
What happens after the tokenizer has done its job?
Let’s suppose a prompt has been converted into 20 tokens. Each token corresponds to a numerical identifier. That identifier is then mapped to a learned vector (an embedding) containing many numerical dimensions. The model does not read the word in the human sense. It operates on these mathematical representations and on their relationships.
A Transformer then uses attention mechanisms to combine information from different positions in the sequence. After processing the context, a generative language model produces a probability distribution over what could come next and selects another token. The process repeats: one output token, then another, then another. Those token IDs are ultimately decoded back into characters that appear on the screen as text.
This is why describing a language model simply as “predicting the next word” is close enough for casual discussion but technically wrong. It normally predicts the next token, and a token can be much smaller or occasionally larger than the word a reader perceives.Nor should a token be mistaken for a unit of thought. A token such as a suffix or punctuation mark does not contain one discrete idea. Its meaning depends on the surrounding sequence and on representations created by the model as it processes that context.
Input, output, cache and reasoning: why there are now several kinds of tokens
Once tokens became a practical measurement unit, providers began using them to describe different stages of a request.
Input tokens are the material supplied to the model. The user prompt, system instructions, relevant conversation history, retrieved documents and, depending on the API, tool definitions and other context. Output tokens are generated by the model.
Cached input tokens are parts of an input that a provider can reuse rather than fully process again when an identical prefix appears in later requests. Prompt caching can therefore reduce cost and computation without increasing the amount of context a model can fundamentally hold.
And reasoning models may consume reasoning tokens while working through a problem before producing the visible answer. Those internal tokens mean that a response consisting of a few sentences can involve considerably more computation than its visible length suggests. OpenAI and Anthropic both account explicitly for reasoning tokens in their current APIs. That makes “How many tokens did this answer use?” a surprisingly complicated question. The text visible on the screen is only part of the transaction.
The same token also measures the model’s working space
Tokens have another job. They measure the size of a model’s context window. The context window is the amount of information available to the model during a request. It can contain instructions, conversation history, source documents, tool results and the response being generated. It should not be confused with the model’s training data or with a product’s long-term memory features.
In September 2026, million-token context windows are already commonplace among top models. OpenAI lists 1.05 million tokens for GPT-5.6 Sol, with up to 128,000 output tokens. Anthropic documents a one-million-token window for Claude Opus 4.8 and several other current Claude models. Google documents a one-million-token input context for Gemini 3 models.
A million tokens sounds enormous. It can make whole codebases, extensive document collections or very long conversations technically possible inside one request. But “fits inside the context window” does not mean “will be used equally well.”
A million tokens of capacity is not a million tokens of perfect attention
This distinction has become increasingly important as vendors compete on context length.
In Lost in the Middle, researchers found that language models could become substantially worse at using relevant information when it appeared in the middle of long contexts rather than near the beginning or end. RULER later tested tasks requiring retrieval, aggregation and multi-step reasoning across longer inputs. Its authors found that models advertising large context windows often suffered substantial performance declines as sequence length and task difficulty increased.
Current models have improved considerably since those early evaluations, but the principle remains important enough that Anthropic’s current documentation explicitly warns that more context is not automatically better and describes degradation at increasing context lengths.
Tokenization also makes some languages more expensive than others
English, French, Spanish, Chinese… each language interacts with a tokenizer differently, often in ways that are far less visible to the user. A 100,000-token context window does not necessarily represent the same amount of human-readable information across every language.
A language that is less efficiently handled by a tokenizer may require more tokens to express the same idea, causing the context window to fill up more quickly. A study published at NeurIPS in 2023 found that, in some extreme cases, translated texts could differ by as much as a factor of 15 in token count depending on the language. The authors highlighted the implications for cost, latency and equitable access to language services. Earlier research had also shown that adapting a tokenizer more closely to a specific language could improve model performance in that language.
It would therefore be misleading to conclude that a prompt written in French will systematically cost far more than an equivalent prompt in English. The difference depends heavily on the tokenizer being used. The broader point is that “one million tokens” is not a perfectly neutral unit when we try to translate it into pages, words or an actual amount of information.
Tokens became money because they roughly track work
Token count is a practical indicator of how much work a model is being asked to perform. For the average consumer using a chatbot, this underlying economy is largely hidden behind a subscription or usage limits. With a professional API, however, it becomes directly visible.
Providers typically distinguish between input and output tokens, and may charge less for cached context. Generated tokens are more expensive because text is produced autoregressively. In other words, the AI generates text piece by piece, with each new token depending on everything that has already been produced. This process is sequential and takes time, whereas much of the input can be analysed and processed simultaneously, in parallel.
For an occasional conversation, saving a few hundred tokens makes virtually no difference. The picture changes for a service processing thousands of documents, or for an agent continuously calling multiple models and tools. Sending the same unnecessary conversation history with every request quickly turns into a recurring cost. Developers therefore have an incentive to retrieve only the most relevant documents, compress older conversations, make use of caching and count tokens before sending a request.
OpenAI now provides a token-counting API capable of estimating the size of a complete request, including messages, files, images and tools. That reflects a broader reality: a rough estimate based solely on character count is no longer enough to capture everything that may contribute to billing.
Tokens have therefore evolved from a linguistic preprocessing choice into a genuine economic unit of the AI cloud.
Images, audio and video complicate the picture further
Modern models no longer consume text alone. Images, audio and video must also be converted into numerical representations the model can process. Providers frequently account for that work using tokens or token-like units, but there is no universal conversion between “one image” and a fixed number of tokens.
Resolution, duration, frame sampling and model architecture can all alter the calculation. Google, for example, documents token accounting across text, images, audio and video, while its video processing modes can produce sharply different token consumption depending on how much of a video the system actually loads.
Do AI models actually need tokens?
While the entire industry now measures context windows and costs in tokens, researchers have spent several years exploring models that could eliminate this step altogether. CANINE, developed at Google, works directly with Unicode characters without relying on a fixed vocabulary, including symbols, emojis and invisible control characters that do not belong to any particular language or alphabet.
ByT5, Google’s tokenizer-free variant of the T5 model, goes one step further by operating directly on the bytes of raw text. Its authors showed that a Transformer could work effectively at this level while remaining competitive across several tasks, with additional gains in robustness to typos and spelling variations.
Meta’s Byte Latent Transformer takes a different approach. Data enters the model as bytes, which are then grouped dynamically into variable-length patches. More complex passages receive finer-grained processing. Experiments published by Meta suggest that this architecture can approach or match the performance of fixed-tokenizer models at scale, while improving certain aspects of efficiency and robustness.
So why have conventional tokens not disappeared yet? Because removing the tokenizer does not remove the underlying problem of sequence length. Raw text represented character by character (or byte by byte) contains far more individual units than text in which common patterns have already been grouped or compressed. That extra length has to be compensated for elsewhere in the architecture.
Tokenizers therefore remain dominant because they provide a practical form of compression before the model begins its most computationally expensive operations. They turn recurring language patterns into larger units in advance, reducing the length of the sequence the model ultimately has to process.
The number behind much of modern AI
Tokens began as a practical answer to a technical problem: how can a neural network process an essentially unlimited language using a finite vocabulary without reducing everything to individual characters? They now occupy a much larger role.
They determine how language is segmented before inference. They measure how much information fits into a model’s working context. They influence latency and multilingual efficiency. They form the unit through which most commercial LLM APIs are priced.
That is why the same term appears in model specifications, developer dashboards and invoices. Understanding tokens does not explain everything a language model does. But it explains the boundary through which almost everything enters and leaves it.