LLM Training vs Inference: What Changes When You Ask a Question

You correct a chatbot's mistake and expect it to remember next time. It won't. You ask the same question twice and get two different answers, which makes the model feel alive—like it's thinking on its feet. But here's the truth that changes how you work with LLMs: the model that answers you is frozen. Nothing about it changes when you talk to it.

The confusion makes sense. The model responds intelligently, so it feels adaptive. But there are two completely different phases in an LLM's life, and almost everything you do happens in only one of them.

Training builds the model. Inference is every time it answers you. That distinction determines what you can fix with a better prompt, what requires retrieval, and what actually needs fine-tuning.

The Confusion: When Does an LLM Actually Learn?

If you've ever typed "remember this for next time" into a chatbot, you've hit the core misconception: that chatting teaches the model.

It feels reasonable. You correct the model, it says "you're right, sorry," and the exchange feels like a lesson. But that apology is just the model generating a plausible response based on patterns it learned long before you arrived. Your conversation is temporary context, not a lesson.

Here's the mental model that fixes everything: an LLM has internal parameters—called weights—that store what it learned. During training, those weights change. During inference, they stay frozen.

Plain language version: training is the phase where the model learns. Inference is the phase where it performs. You interact with inference constantly. You almost never touch training.

Training: The Expensive Phase That Builds the Model

Training is how AI models learn. The model is exposed to enormous amounts of text—books, articles, code, conversations—and its weights are adjusted over and over until it gets good at predicting what comes next in a sequence.

Think of training like a student studying for months before an exam. The studying happens once, in private, before the test day arrives. The student builds knowledge through repeated exposure and correction. When the exam comes, they don't get to study anymore—they just answer.

That's training: a one-time or occasional investment. It requires massive GPU clusters, huge datasets, and long runtimes. We're talking weeks of compute and costs that reach millions of dollars for the largest models.

Here's the practical reality for you: you will almost certainly never train an LLM from scratch. The models you use—through APIs, chatbots, or open-source weights—were trained by someone else. Your job starts after training ends.

Inference: What Happens When You Ask a Question

LLM inference is what happens at runtime, every time you send a request. The trained model takes your input and generates a response, one token at a time, predicting each next piece of text based on patterns it learned during training.

Back to the exam analogy: inference is the student answering questions. All that knowledge built during study gets applied to new problems. No new studying happens. The student doesn't get smarter while taking the test.

The key fact: during inference, the model's weights are frozen. Nothing about the model changes. Every chat message you send, every email draft you generate, every code suggestion you accept—each one is an inference call.

Inference happens continuously and in real time. It's the phase where the model earns its keep, answering one request after another. And while each individual call is cheaper than training, those costs add up fast across millions of requests.

Training vs Inference: The Side-by-Side

The memorable one-liner: training is where the model learns; inference is where it earns its keep.

Four concrete tasks make the split obvious:

• Answering a chat message — inference. The frozen model reads the conversation and generates a reply.
• Fine-tuning on support tickets — training. The model's weights are adjusted so it answers in your company's voice.
• Generating an image from a prompt — inference. A trained model turns text into pixels.
• Pretraining a base model on web text — training. Weights are built from scratch over weeks on large clusters.

Fine-Tuning: Training in Disguise

Beginners often misclassify fine-tuning as something that happens while you chat. It doesn't. Fine-tuning is a form of training.

When you fine-tune a model, you update some of its weights using new data to adapt it to a specific task or domain. Say you run a support team and want the model to answer in your company's voice, with your product knowledge. You fine-tune it on your support tickets. That's a deliberate project—you're changing the model itself.

The contrast with inference is sharp: fine-tuning changes the model; inference only uses it. Fine-tuning is an occasional, intentional investment. It doesn't happen by accident while you're having a conversation.

What Your Prompt, Retrieval, and Fine-Tuning Actually Change

Here's the payoff—a decision rule you can use every day.

Think of an LLM system as having layers:

1. The frozen model — the trained weights that never change during use
2. The prompt — what you send for a single request
3. Retrieved context — additional information added to your prompt from external sources

Your prompt and retrieval change only the input for one inference call. They never touch the model's weights. You're giving the frozen model better material to work with, not making it smarter.

Fine-tuning and new training change the model itself. They affect every future inference call, for everyone who uses that model.

The decision rule:

• Want a different answer for one request? Change the prompt or add retrieval.
• Want the model to behave differently for everyone, permanently? That requires training or fine-tuning.

Let's test it with a scenario. Your support model keeps giving generic answers to customer questions. The immediate fix: improve your prompt with better instructions, or add retrieval so the model can pull from your knowledge base. That solves the case at hand. But if you want the model to inherently understand your product's tone and common issues—for every customer, every time—you fine-tune it on your support data.

Common Beginner Mistakes and How to Recover

Mistake 1: Believing the model learns from your conversation. Your chat history is just context for one inference call. The model forgets everything when the conversation ends.

Recovery: If you need consistent behavior across sessions, save important context and resend it in future prompts. If you need the model to permanently understand something, that's a fine-tuning conversation.

Mistake 2: Assuming fine-tuning is the only way to improve answers. Fine-tuning is powerful, but it's also expensive and slow. Prompt design and retrieval often deliver faster wins.

Recovery: Before investing in fine-tuning, test whether better prompts or adding relevant context through retrieval solves your problem. For many use cases, it does.

Mistake 3: Confusing the cost model. Training is expensive once. Inference is cheap per call but adds up across millions of requests.

Recovery: Budget for both. A single training run is a big bill. But if you're serving thousands of users, your ongoing inference costs will likely exceed that training bill over time.

The Rule That Covers Everything

When you want a different answer, change the prompt or retrieval. When you want the model itself to change, that's training or fine-tuning.

That one sentence tells you which lever to pull in almost any situation. It also tells you what not to waste time on: trying to teach a model through conversation, or fine-tuning when a better prompt would do.

The natural next step is understanding how much context you can give the model in a single inference call—because that limit shapes what your prompts and retrieval can actually accomplish.