Next-Token Prediction Explained: How LLMs Generate Text

When you watch a large language model type out a response, it feels like the system thought through the whole answer before writing a single word. It didn't. An LLM writes the way a phone keyboard suggests the next word—one small prediction at a time—just with vastly more context and training behind each choice.

The difference between how an LLM feels and what it actually does is the single most useful mental model you can build. Once you see the one-token-at-a-time mechanism underneath the fluent output, a lot of otherwise confusing behavior starts to make sense: why long answers drift, why the model sounds confident even when it's wrong, and why the words you put in matter so much.

The One-Token-at-a-Time Trick

Here's the core idea: an LLM does not write from a finished plan the way a human essayist does. It predicts the single most likely next piece of text, adds that piece to what it has so far, and then predicts again from the new, longer context.

Think about your phone's autocomplete. When you type "I'm going to the," your keyboard suggests one next word: "store," "gym," "park." It doesn't draft the rest of your sentence for you. It makes one small guess based on what you've typed so far.

A large language model works on the same principle, but with two enormous upgrades. First, it has seen billions of examples of human text during training, so its guesses are far more sophisticated. Second, it doesn't just predict one word for your whole message—it predicts the next piece of text, then the next, then the next, building a response one step at a time.

The surprising part is that this simple loop produces text that looks genuinely planned. Coherent paragraphs, logical arguments, even jokes that land—all of it emerges from thousands of tiny, locally sensible choices stacked on top of each other.

Before we go further, one quick note on vocabulary. LLMs don't actually predict whole words. They predict tokens, which are the text units the model works with. A token might be a whole word like "cat," a piece of a word like "un" or "ing," or even a single punctuation mark. For this article, you can think of tokens as the building blocks the model reads and writes, one at a time.

Let's use a running example throughout this article. Imagine you type this into an LLM:

The cat sat on the

What happens next inside the model is the heart of next-token prediction.

From Context to a Probability Distribution

When the model receives your text, it doesn't search for a "correct" continuation. Instead, it performs a kind of vote across everything it could possibly say next.

Every LLM has a fixed vocabulary—a list of every token it knows how to output. For many models, that's tens of thousands of possibilities. Given the context you've provided, the model assigns a probability to every single token in that vocabulary.

A probability distribution sounds technical, but here's what it actually means: the model spreads a total of 100% probability across all the tokens it knows, giving more to the ones that fit the context and less to the ones that don't. The ranked list you see is just the top of that distribution.

After "The cat sat on the," the distribution might look something like this:

Those percentages are illustrative, not the model's actual numbers—but the shape is real. The model has learned these patterns from the enormous amount of text it saw during training. It has encountered countless sentences about cats sitting on mats and floors, and almost none about cats sitting on computers. Those statistical patterns are what push certain tokens to the top of the distribution.

This is also why context matters so much. Change the prompt to "The programmer sat on the," and the distribution shifts dramatically. "Chair" and "desk" climb to the top, while "mat" drops. The model isn't consulting a rulebook about programmers or cats. It's using the context you gave it to shape which tokens seem likely next.

One important thing to notice: the model isn't checking whether a continuation is true. It's checking whether a continuation is statistically likely based on patterns in its training data. That distinction will matter again later.

Choosing the Next Token: Greedy vs. Random

So the model has its probability distribution. Now it needs to pick one token to actually output. This is where things get interesting, because the model doesn't always pick the most likely option.

Greedy decoding is the simplest strategy: always choose the token with the highest probability. After "The cat sat on the," the model picks "mat," every time.

Greedy decoding sounds like the obvious choice, but it has a problem. If the model always takes the most probable path, its responses become flat, repetitive, and predictable. Think about how a phone keyboard's top suggestion can get annoying—now imagine an entire conversation built from only the most obvious next word.

Sampling is the alternative. Instead of always picking the top token, the model chooses randomly, but with higher-probability tokens more likely to win. It's like a weighted lottery: "mat" gets 42 tickets, "floor" gets 18, and "xylophone" gets almost none. Most of the time the model picks a sensible token, but occasionally it picks something slightly less obvious.

That randomness is what makes LLM output feel natural and varied. Ask the same question twice with sampling enabled, and you'll get two different but equally valid responses. The model isn't being inconsistent—it's exploring different paths through its probability distribution.

Many LLM interfaces expose a temperature setting, which is a dial on this randomness. Lower temperature makes the model behave more like greedy decoding: predictable, focused, and a bit dull. Higher temperature makes the model more adventurous: varied and creative, but more likely to go off the rails.

My practical rule for beginners: if you need consistent formatting or a repeatable style, keep the temperature low. If you're brainstorming or want creative variety, let it run warmer. Just remember what temperature actually changes: how boldly the model picks among what it considers possible. It doesn't verify facts or repair missing knowledge. A low temperature can make a wrong answer repeatable—it just can't make that answer right.

The Loop: Append, Predict, Repeat

Now we can put the whole process together. Generating a response isn't one prediction—it's a loop that repeats until the model decides to stop.

Let's trace our example step by step:

Step 1. Context: "The cat sat on the" The model produces a probability distribution and selects "mat."

Step 2. Context: "The cat sat on the mat" The model produces a new distribution and selects "and."

Step 3. Context: "The cat sat on the mat and" The model produces another distribution and selects "watched."

Step 4. Context: "The cat sat on the mat and watched" The model selects "the."

Step 5. Context: "The cat sat on the mat and watched the" The model selects "bird."

Each step, the model's own previous output becomes part of the context for the next prediction. This is what makes the model autoregressive: it feeds on its own outputs, using what it just wrote to decide what to write next.

This loop also explains why generation can feel slow. Every token is a separate prediction step, and each step builds on the full context so far. A long response might require hundreds or thousands of sequential steps. That's why a 500-word answer takes noticeably longer than a 50-word one—the model isn't writing faster; it's doing more prediction cycles.

How does the model know when to stop? Two mechanisms work together. The model can predict a special end-of-sequence token when it judges the response complete—the equivalent of typing a period and putting down the pen. The system can also enforce a maximum length limit as a safety net. Either way, the loop ends, and the response you see is the accumulated result of every individual prediction.

Why This Changes How You Should Think About LLMs

Understanding next-token prediction isn't just academic curiosity. It explains several behaviors that confuse beginners.

The model generates without a separately exposed master plan. It commits to each token before the full response exists. That doesn't mean the model is incapable of structure—its training has built patterns that can carry repetition, callbacks, and long-range organization through the context. But nothing guarantees the whole response hangs together, which is why long outputs can drift, repeat themselves, or contradict something stated a few paragraphs earlier. The earlier text constrains the later text, but coherence has to survive thousands of sequential decisions.

Clear prompts produce better predictions. Since every prediction depends on context, the quality of your input directly shapes the quality of the output. A vague prompt gives the model a fuzzy distribution, where many different continuations seem plausible. A specific prompt narrows that distribution, making the kind of response you want more likely. You're not "telling the model what to think"—you're giving it better context to predict from. A clearer prompt makes the desired continuation more probable, but it still doesn't guarantee correctness.

Confidence is not accuracy. The model can sound utterly certain while being completely wrong, because it's choosing the statistically likely next token, not verifying a fact. "The capital of France is" produces a high probability for "Paris," but the model isn't checking a database—it's recalling a statistical pattern. Most of the time those patterns align with reality. When they don't, you get confident nonsense.

Treat long generations as drafts. Because coherence has to survive thousands of sequential decisions, long responses deserve a skeptical read. Check for internal consistency, verify important claims, and be ready to ask for revisions. The model isn't delivering a finished document—it's delivering a very sophisticated first pass.

Here's the common beginner mistake I see: assuming the model decided the whole answer up front, then feeling confused when it contradicts itself or misses an obvious point. Once you internalize that each token was a fresh local prediction built on everything before it, those failures stop being mysterious. They're the natural consequence of the mechanism.

Putting the Mental Model to Work

The best way to make this stick is to watch it happen. Open any LLM and give it a short, incomplete phrase like "The best thing about winter is." Then watch the response build. Notice that the model isn't delivering a pre-written paragraph—it's committing to each token before the next one exists.

If you want to see the mechanism even more clearly, ask the model to complete a phrase one word at a time, or use a model interface that shows tokens streaming in as they're generated. You'll see the response assemble itself piece by piece, each token a fresh prediction built on everything before it.

The one-sentence takeaway: an LLM is a very good one-step-at-a-time predictor, not a planner with a finished draft in hand. It makes the next token as likely as possible, over and over, until the response is done.

That raises an obvious question: how does the model actually compute those probabilities? What's happening inside during each prediction step? That's where the transformer architecture comes in—the mechanism that lets the model weigh which parts of your context matter most for the next token. Understanding next-token prediction gives you the what; the transformer explains the how.