beginner
8 min read
Estimate the Memory a Local LLM Needs
Two people run the same 7B model. One says it needs 4 GB. The other says it needs 9 GB. Neither is lying.
Read tutorialconcept
Model Inference
How a trained model is used at runtime to process context and generate outputs, including decoding and inference resource behavior.
Tagged articles
Model Inference articles
18 articles in this tag.
intermediate
9 min read
From Next-Token Probabilities to Autoregressive Sequence Probability
A language model never scores a sentence. It scores one token at a time — and the sentence score is just those pieces multiplied together.
Read tutorial
beginner
11 min read
How Do LLMs Work?
Large language models are not digital minds. They are probability engines that turn a conversation into a series of next-token guesses. The guesswork is…
Read tutorial
advanced
9 min read
Inference-Time Compute: Reasoning About Extra Model Effort
A builder doubles the candidate count, watches the benchmark number climb, ships it, and then discovers that p95 latency and per-request cost moved in ways…
Read tutorial
beginner
11 min read
What to Learn Before LLM Frameworks: A First-Principles Path
Frameworks change. The model underneath them doesn't. Learn the mechanism first, and every new tool becomes just another wrapper around something you…
Read tutorial
intermediate
7 min read
LLM Batching and Throughput: Why Interactive Speed Is Not the Whole System
A single fast response tells you almost nothing about whether your LLM system can handle real load. The request that feels snappy in isolation is often the…
Read tutorial
intermediate
9 min read
LLM Batching and Throughput: Work Through the Numbers
Throughput climbs, the dashboard looks healthy, and p95 quietly doubles. Here is the arithmetic that explains why.
Read tutorial
beginner
12 min read
LLM Generation Settings Explained: Temperature, Top-P, and Token Limits
You asked for a summary and got a rambling essay. You asked for a story and got three bullet points that stopped mid-thought. You asked a factual question…
Read tutorial
advanced
9 min read
LLM KV Cache Memory: Work Through the Inference Math
The weights fit. The model loads. Then the second concurrent request arrives, and the server refuses to admit it.
Read tutorial
beginner
10 min read
LLM Temperature Explained: What It Changes and What It Cannot Fix
You turned the temperature down to 0.1 because you wanted the AI to stop making mistakes. It still gave you a wrong answer—just a more confident, more…
Read tutorial
beginner
7 min read
LLM Training vs Inference: What Changes When You Ask a Question
You correct a chatbot's mistake and expect it to remember next time. It won't. You ask the same question twice and get two different answers, which makes…
Read tutorial
beginner
10 min read
Next-Token Prediction Explained: How LLMs Generate Text
When you watch a large language model type out a response, it feels like the system thought through the whole answer before writing a single word. It…
Read tutorial
beginner
11 min read
Practice Diagnosing LLM Output Problems from Generation Settings
The output looks wrong, and your hand is already on the temperature slider. Before you drag it, notice what you are actually doing: guessing. The same…
Read tutorial
beginner
9 min read
Run a Small Local LLM and Measure Its Resource Use
The model finishes downloading. You type a prompt. The cursor blinks, the fan spins up, and nothing happens for a while. That pause is the whole question:…
Read tutorial
intermediate
11 min read
Temperature and Token Sampling: The Probability Math
Temperature does not add randomness to a model. It reshapes a probability distribution the model already produced — and once you see the arithmetic, you…
Read tutorial
beginner
11 min read
Transformer Architecture Explained: The Mechanism Behind Modern LLMs
You know an LLM predicts the next word. The real question is how it decides which word deserves to come next—and the answer lives in a design called the…
Read tutorial
absolute beginner
7 min read
What Is a Large Language Model?
A large language model is a pattern-prediction engine for language: it learns how words and ideas tend to follow one another, then uses that fluency to…
Read tutorial
beginner
11 min read
Why Do LLMs Hallucinate? A Practical Mental Model
You ask a chatbot for a quick fact. The answer comes back in clean, confident sentences—specific dates, plausible names, a citation that looks real. It…
Read tutorial