How LLM Tokenization Works: Why Words Become Different-Sized Pieces

You type a sentence into an AI chat window. The model reads it, thinks, and answers back. Somewhere in that exchange, your words get chopped into pieces—and the cuts don't always land where you'd expect. A word you'd naturally treat as one unit might split into three. A phrase that feels long might compress into a single piece.

Most people assume tokens roughly equal words. That assumption breaks the moment you look at real output. Here's what's actually happening: before an LLM reads anything, a separate system called a tokenizer cuts your text at boundaries you never chose. Those cuts determine how much context your prompt fills, how much it costs, and how the model processes what you gave it.

Why "One Word = One Token" Is Wrong

Your default model probably looks like this: a token is basically a word, maybe with punctuation attached. "The cat sat" becomes three tokens. Simple.

Now watch where that model fails. The word "the" might be one token—it appears constantly in training data, so the tokenizer learned to keep it whole. But a rarer word like "unpredictability" might split into several pieces: "un," "predict," "ability," or something similar. Same grammatical category. Very different token count.

The tokenizer isn't following grammar rules. It's using a fixed lookup table—a vocabulary of pieces it learned during a separate training pass. That table was built by analyzing enormous amounts of text and finding the most efficient cuts. The result is that token boundaries follow statistical frequency, not linguistic structure.

Here's the reframe that matters: tokenization cuts text at boundaries the model learned to recognize, and you did not choose where the cuts land. You supply the words. The tokenizer decides how to slice them.

The Tokenizer as a Fixed Set of Building Blocks

Think of the tokenizer as a keyboard with a fixed set of keys. The model can only press keys that exist on that keyboard. Every key corresponds to a piece of text—a common word, a word fragment, a single character, or a special symbol—and each piece maps to a number.

When you send a prompt, here's the sequence:

1. The tokenizer normalizes your text (handling spacing, punctuation, and special characters).
2. It looks up your text against its vocabulary and cuts it into the best-matching pieces.
3. Each piece becomes a numeric ID.
4. The model reads that sequence of numbers—never your raw letters.

The tokenizer sits outside the model. It converts text to IDs before the model runs, and converts IDs back to text afterward. The model itself works entirely in numbers.

A Lego analogy works well here. The tokenizer's vocabulary is your brick collection. You can build almost anything, but only from the pieces you own. If you don't have a single brick shaped like "unpredictability," you assemble it from smaller bricks you do have. The model never sees the original word—only the assembled structure of whatever pieces the tokenizer chose.

This is also why tokens are the billing unit for commercial LLM APIs. You're paying for the number of pieces the tokenizer produced, not the number of words you wrote.

Why Subword Pieces Beat Whole Words

If whole-word tokens are misleading, why not just use them properly? Because that approach collapses under real-world text.

Imagine a tokenizer that keeps every word whole. English alone has hundreds of thousands of words, and new ones appear constantly—names, technical terms, slang, misspellings. The vocabulary would balloon to an unmanageable size, and any word the tokenizer hadn't seen during training would be unrepresentable. The model would choke on anything unfamiliar.

The opposite extreme—one token per character—solves the unfamiliar-word problem but creates a new one. Every word becomes a long sequence of tiny pieces. "Hello" becomes five tokens. A paragraph becomes hundreds. The model has to work much harder to assemble meaning from fragments, and it loses the patterns that make frequent words efficient to process.

Subword tokenization splits the difference. Frequent words stay whole. Rare or new words break into smaller pieces the tokenizer has seen before. The word "unhappiness" might split into "un," "happi," and "ness." The tokenizer learned these boundaries by analyzing which piece-combinations appeared most often across massive text corpora.

Here's a concrete example. Paste the word "tokenization" into a tokenizer tool for a popular model, and you'll likely see it split into pieces like "token" and "ization"—or even "tok," "en," and "ization," depending on the model. The same word can split differently across models because each model's tokenizer learned its own vocabulary from its own training data.

This is why the same spelling can be one token in one model and several in another. It depends entirely on what that model's vocabulary contains. Algorithms like Byte Pair Encoding (BPE) and WordPiece handle the mechanics, but you don't need their internals to understand the result: frequent text compresses, unfamiliar text expands.

Why Token Counts Vary for the Same Text

Here's where the mental model becomes practically useful. Since token boundaries follow learned frequency, you can predict which text will cost more tokens.

Common words and phrases compress into few tokens. Rare words, unusual spellings, technical jargon, and unfamiliar names expand. The effect is dramatic across languages: a language well-represented in the model's training data might average close to one token per word, while a less-represented language might need several tokens for the same meaning.

Consider two sentences with identical character counts:

• "The weather today is beautiful and warm." — mostly common words, likely close to one token per word.
• "The baroreceptor reflex modulates chronotropic response." — rare technical terms, likely two or three tokens per word.

Same length on screen. Very different token counts. That difference determines how much of the model's context window your text occupies and how much the request costs.

A useful way to visualize this: imagine color-coding a sentence by token boundaries. Some words stay whole—one color, one piece. Others split mid-word, with each fragment a different color. The pattern looks nothing like the spaces between words.

Note: The same word can also tokenize differently when its neighbors change. "Tokenization" followed by a space may split one way; "tokenization." with a period attached may split another. The tokenizer isn't responding to meaning—it's responding to the exact characters it sees, including whitespace and punctuation.

What Token Boundaries Mean for You

This mental model stops being academic the moment you build something with an LLM. Three practical consequences follow directly from how tokenization works.

Context limits are measured in tokens, not words. Every model has a maximum context window—the total text it can consider in one request. Since tokenization is content-dependent, the same character count can fill very different amounts of that window. A prompt full of rare terms or unfamiliar languages consumes more context than the same-length prompt in common English. When you're estimating whether a document fits, word count is a poor proxy. Token count is the real measurement.

Cost scales with token count. Commercial LLM APIs charge per token, for both input and output. Text that tokenizes inefficiently costs more per request. If you're building an application that processes large volumes of text, tokenization differences directly affect your operating costs. Verbose or unusual text is quietly more expensive.

The model generates one token at a time. When a word splits into several pieces, the model must predict each piece sequentially. More pieces mean more steps—and longer generation time. That's a measurable cost. Whether those extra steps also affect output quality is a separate question. Tokenization changes the sequence the model processes, but it doesn't act alone. The model's training, the surrounding context, and the model's own reasoning all shape the final answer.

The practical rule: when you need to estimate length or cost, measure with the exact tokenizer the model uses. Most model providers offer tokenizer tools where you can paste text and see exactly how it splits. Guessing by word count will mislead you.

Where This Mental Model Helps and Where It Stops

The tokenization model is powerful for predicting token counts, understanding cost differences, and explaining why some words behave oddly. When a model struggles with an unusual name or a less-common language, tokenization is often a contributor—the text may consume more of the context window or break into pieces the model handles less efficiently.

But it's not the whole story. Tokenization is one input to model behavior. Meaning, context, training data, and the model's architecture all shape output quality. If a model gives a wrong answer on a straightforward question, the cause is probably not token boundaries—it's the model's reasoning or knowledge. Don't over-attribute mistakes to tokenization when the real cause lives elsewhere.

The model also stops being useful for predicting quality directly. A word that splits into more tokens isn't automatically handled worse. What matters is whether the pieces give the model a workable representation—and that's something you can only judge by testing real outputs, not by staring at word fragments.

Here's your next step. Open a tokenizer tool for whatever model you use most. Paste a sentence you actually care about—a prompt you've used, a document you're processing. Watch where the cuts land. Notice which words stay whole and which split. Then estimate the token cost of a real prompt you use regularly.

The durable takeaway: token boundaries are learned cuts, not grammar. Once you see text the way the tokenizer slices it, you can predict context usage, estimate costs, and understand why some inputs behave differently than your intuition expects. The cuts you never chose are shaping how your words enter the model—and that's worth measuring, not guessing.