Transformer Architecture Explained: The Mechanism Behind Modern LLMs

You know an LLM predicts the next word. The real question is how it decides which word deserves to come next—and the answer lives in a design called the transformer architecture.

Here's what surprises most beginners: transformers don't read text the way you do. They don't march through a sentence word by word, building meaning step by step. Instead, they process many tokens at once, let each one gather context from the tokens it's allowed to see, and refine those representations through layer after layer. By the end of this article, you'll be able to trace what happens from the moment text enters a model to the moment it outputs a prediction.

Why Transformers Replaced Earlier Language Models

Before transformers, language models worked like someone reading a book one word at a time and trying to remember everything they'd seen so far. These earlier models, called recurrent neural networks, processed text sequentially. Each word was fed in order, and the model updated its internal state as it went.

That approach had two serious problems. First, it was slow—processing words one at a time meant the model couldn't take advantage of modern hardware designed to do many calculations at once. Second, these models had a terrible memory. When a word appeared early in a sentence and the information needed to understand it appeared much later, the model often lost the thread. The longer the gap between related words, the more likely the connection would fade.

The transformer, introduced in the 2017 research paper "Attention Is All You Need," solved both problems with one core idea: process many tokens simultaneously, and let each token directly gather information from the other tokens around it. No strict sequential reading. No fading memory. Just a direct line between related tokens, no matter how far apart they sit.

That shift is why transformer-based designs now power essentially every major LLM you've encountered. When you ask a chatbot a question, a transformer is doing the work.

The Big Picture: Tokens In, Next Token Out

Let's build a mental map before we zoom into the parts.

When you send text to an LLM, here's what happens at the highest level:

1. Your text gets broken into small pieces called tokens.
2. Each token enters the transformer as a list of numbers.
3. Those numbers pass through many processing layers stacked on top of each other.
4. The final layer produces a probability for every possible next token.
5. The model picks one, and the process repeats.

The transformer's job during generation is deceptively simple: predict the next token, one at a time. But that simple job, repeated thousands of times, produces everything from email drafts to code to poetry.

The key image to hold onto: a transformer is a stack of layers, and each layer refines the representation of every token based on the tokens around it. Think of it like a group conversation where each participant hears the people who have already spoken, then adjusts what they say next based on what they heard. Now imagine that conversation happening dozens of times, with each round producing more nuanced understanding.

One important note before we go deeper: a token isn't always a whole word. It can be a word, part of a word, punctuation, or a special marker. The architecture works the same way regardless—so from here on, I'll use "token" when we're talking about what the model actually processes.

Turning Text Into Numbers: Embeddings and Position

Here's the first hard truth about language models: they can't read. Letters, words, sentences—none of it means anything to a model until it becomes numbers.

The conversion happens through embeddings. Each token gets mapped to a long list of numbers that represents its meaning. The token "cat" becomes a list of perhaps thousands of numbers, each capturing some dimension of what "cat" means. The token "dog" gets a different list, but one that's similar in certain ways because cats and dogs share conceptual territory.

Here's a crucial distinction: the initial embedding is a starting point, not the final meaning. The token "bank" needs a representation that can shift meaning between "river bank" and "money bank." The embedding provides the raw material, but the transformer's later layers do the work of adjusting that representation based on the surrounding context.

There's a second problem the transformer must solve: order. Because the model processes many tokens at once, it has no built-in sense of sequence. The sentence "cat chased dog" and "dog chased cat" contain the same tokens, but they mean opposite things. So the transformer adds position information to each token's embedding—a signal that says "I'm the first token" or "I'm the third token."

You don't need the math here. Just remember: every token carries two kinds of information into the transformer—what it means and where it sits in the sequence.

Attention: How the Model Decides What Matters

Now we reach the heart of the transformer: the attention mechanism.

Here's the question attention answers: for each token, which other tokens should influence its meaning?

Consider this sentence: "The cat sat on the mat because it was tired."

What does "it" refer to? A human reader instantly knows—the cat. But a model has to figure that out from the patterns it learned. Attention is the mechanism that lets it do so. The token "it" looks at the other tokens it's allowed to see and asks, "How relevant are you to understanding me?" The token "cat" gets a high relevance score. The token "mat" gets a lower one. The token "the" gets almost none.

Now, here's the part that trips up most beginners: in the kind of transformer used by modern generative LLMs, each token can only look at earlier tokens—not future ones. When the model is predicting what comes after "The cat sat on the," the token at that position can attend to "The," "cat," "sat," "on," and "the"—but it cannot see a word that hasn't been generated yet. That restriction is called a causal mask, and it's what makes next-token prediction honest. The model can't cheat by peeking ahead.

This doesn't mean the model is slow. During training, when all the tokens in a sentence are already known, the model can process many positions in parallel—each one attending only to earlier positions, but all of them computed at the same time. During generation, the newly predicted token gets appended, and the process repeats.

So the conversation metaphor needs one adjustment: it's not a room where everyone hears everyone else. It's a room where each speaker hears the people who spoke before them, then adds their own contribution. The meaning of "it" in our example can be shaped by "cat" and "sat" and "mat"—all of which came earlier—but not by a word that hasn't been spoken yet.

This happens for every token position. Each one looks at the earlier tokens, assigns a relevance weight, and gathers information from the tokens that matter most. The token "sat" pays attention to "cat" because cats are the things that sit. "Tired" connects back to "cat" for the same reason. Every token ends up with a richer representation because it has pulled in context from its permitted neighbors.

Now multiply that by a concept called multi-head attention. Instead of asking one "which tokens matter?" question, the model asks several at once, each looking for a different kind of relationship. One attention head might track grammatical connections. Another might track pronoun references like our "it" to "cat" example. Another might track whether tokens are related in meaning. Running these questions in parallel gives the model multiple perspectives on the same text simultaneously.

Here's the intuition to keep: attention is the model's way of saying, "To understand this token properly, I need to know what else is here—specifically, what came before it."

Stacked Layers: Refining Meaning Step by Step

One pass through attention isn't enough. The model needs to refine its understanding iteratively, which is why transformers stack many layers.

Think of each layer as another read of the text with better context. The first layer catches basic relationships—which tokens are near each other, which tokens modify which. The second layer builds on that, catching subtler connections. By layer ten or twenty, the model has assembled a rich picture of the text's meaning that no single pass could achieve.

Each layer does two things. First, it runs attention, letting every token gather context from the tokens it's allowed to see. Second, it runs a processing step called a feed-forward network that lets each token transform its own representation independently. You can think of attention as the conversation part—tokens sharing information—and the feed-forward step as the reflection part—each token deciding what to do with what it learned.

Modern LLMs stack many of these layers. The depth matters. Shallow models capture shallow patterns. Deep models, with layer after layer of refinement, can capture the kind of complex linguistic structure that makes their output feel genuinely intelligent.

From Final Layer to Next-Token Prediction

After the last layer, every token position holds a refined representation—a list of numbers that captures both what the token means and how it functions in this specific context.

The final step is a kind of vote. The model takes the representation of the last token position—the one where the next token will appear—and compares it against every token in its entire vocabulary, which can be tens of thousands of possibilities. Each vocabulary token gets a probability score. The model then picks the token with the highest probability, and that becomes its prediction.

Let's make this concrete. Suppose the model is generating text and has produced "The cat sat on the." The final token position holds a representation that encodes everything the model has learned from that prefix. When it compares that representation against its vocabulary, tokens like "mat," "floor," and "couch" receive high probabilities. Tokens like "airplane" or "democracy" receive near-zero probabilities.

The model doesn't "choose" a word the way you might choose a word. It ranks every token it knows by likelihood and selects the top candidate. That ranking is the output of the entire transformer architecture—the final product of embeddings, attention, and many refinement layers.

Common Misconceptions About Transformers

Let me clear up three mental models that trip up beginners.

Misconception 1: "The model reads the text and understands it like a person."

It doesn't. The transformer performs weighted pattern matching over numbers. When it connects "it" to "cat," it's not having an insight about feline pronouns. It has learned statistical patterns from billions of sentences that tell it "it" near "cat" and "tired" probably refers to the cat. The output looks like understanding, but the mechanism is pattern recognition at massive scale.

Misconception 2: "Attention means the model remembers everything."

Attention works within the current context window—the text the model can see right now. It's not long-term memory. When you start a new chat session, the model doesn't remember the previous conversation unless that conversation is included in the current context. Attention is a spotlight, not a storage system.

Misconception 3: "The transformer is one big model."

"Transformer" names an architecture, not a specific product. Think of it as a blueprint. Different companies use that blueprint to build different models with different sizes, training data, and design choices. Many modern LLMs use transformer-based designs, but they're distinct models built from the same fundamental pattern—and some vary the details.

Putting Your Mental Model to Work

You now have the full journey: text becomes tokens, tokens become embeddings, embeddings gain position information, attention lets each token gather context from the tokens before it, stacked layers refine meaning step by step, and the final layer produces a probability ranking over the entire vocabulary.

Here's a way to test your understanding. Take the prefix "The chef cooked the meal because she was." Before you send it to any LLM, ask yourself: what can the model see at this point? It can see every token in that prefix—including "she," which needs to connect back to "chef." It cannot see whatever comes next, because that token hasn't been generated yet. When the model predicts the next token, it's ranking candidates like "hungry" or "tired" against everything else in its vocabulary, based only on what the prefix tells it.

Now try it. Send that prefix to any LLM and see how it continues. You're not just observing output anymore—you're watching the prediction step you now understand, one token at a time.

The natural next step is understanding tokens more deeply, since they're the fundamental unit everything else builds on. But for now, you've opened the black box. The transformer isn't magic anymore. It's an engine, and you know what its parts do.