intermediate
12 min read
Agent Loop Budgets: Model Iterations, Cost, and Stopping Risk
A loop that usually finishes in three steps can still spend most of its money on the runs that don't.
Read tutorialworkflow
LLM Deployment
Selecting, integrating, and operating model services and LLM application infrastructure under practical resource and service constraints.
Tagged articles
LLM Deployment articles
26 articles in this tag.
intermediate
9 min read
Calculate the Economics of LLM Cache Hits
A 60% hit rate does not mean a 60% discount. It means you have one number and three missing ones.
Read tutorial
beginner
9 min read
Cloud vs Local LLMs: Which Setup Fits Your First Project?
The "local is more private, cloud is more powerful" story sounds clean. It's also too simple to make a good decision with.
Read tutorial
beginner
10 min read
Make Your First Programmatic LLM API Request
The four-line API call is not the hard part. The hard part is the key, the bill, and the error message you cannot read. So let's get all three out of the…
Read tutorial
intermediate
11 min read
Hosted vs Local LLM Cost: Calculate the Break-Even Workload
Two people run the same break-even calculator, feed it the same hardware price, and get opposite answers. Both are right.
Read tutorial
advanced
9 min read
Inference-Time Compute: Reasoning About Extra Model Effort
A builder doubles the candidate count, watches the benchmark number climb, ships it, and then discovers that p95 latency and per-request cost moved in ways…
Read tutorial
intermediate
9 min read
LLM API Rate Limits: Calculate Request and Token Capacity
At 9:05 a.m., your dashboard says you are fine. Your daily average sits comfortably inside the published limits. Then the 429s start.
Read tutorial
intermediate
7 min read
LLM Batching and Throughput: Why Interactive Speed Is Not the Whole System
A single fast response tells you almost nothing about whether your LLM system can handle real load. The request that feels snappy in isolation is often the…
Read tutorial
intermediate
9 min read
LLM Batching and Throughput: Work Through the Numbers
Throughput climbs, the dashboard looks healthy, and p95 quietly doubles. Here is the arithmetic that explains why.
Read tutorial
intermediate
9 min read
LLM Caching Explained: Reducing Cost Without Serving Stale Answers
The invoice climbs the moment your prototype starts running regularly. The same system prompt is reprocessed from scratch on every call. The same FAQ is…
Read tutorial
intermediate
9 min read
LLM Cost, Latency, and Quality Tradeoffs Explained
Most early builders ask which LLM is the smartest. That is the wrong first question. The right question is: what does your feature actually need, request…
Read tutorial
advanced
9 min read
LLM KV Cache Memory: Work Through the Inference Math
The weights fit. The model loads. Then the second concurrent request arrives, and the server refuses to admit it.
Read tutorial
intermediate
12 min read
LLM Model Routing Explained: Choosing the Right Model Per Task
Here's a scenario I see constantly: an application sends every request to one frontier model. Summarize this email in two sentences? Frontier model.…
Read tutorial
intermediate
10 min read
LLM Routing Workflows: Classify the Task Before Choosing a Path
When builders hit a multi-path LLM application, many reach for an agent. That instinct costs them. The real mechanism for most of these systems is a…
Read tutorial
intermediate
13 min read
How to Aggregate LLM Latency and Usage: A Worked Example
Your dashboard says the average response takes 1.2 seconds. Your users say the app feels slow. Both statements are true, and that is the problem.
Read tutorial
intermediate
10 min read
Model the Cost and Stale-State Risk of Agent Memory
A team sets memory to refresh every six hours to save money. Four hours later, an agent quotes a price, policy, or preference that has already changed. The…
Read tutorial
intermediate
12 min read
LLM Model Routing Break-Even Analysis: Work Through the Math
Routing is a bet on a classification. The break-even tells you the odds before you place it.
Read tutorial
beginner
13 min read
Open-Weight vs Closed LLMs: What Control Do You Actually Need?
You do not need the best model. You need the model that matches the control you actually require.
Read tutorial
beginner
8 min read
Practice Choosing an AI Tool from Realistic Constraints
Most beginners pick an AI setup the way they pick a restaurant: the one that sounds best, then hope the bill and the wait are tolerable. Then the…
Read tutorial
beginner
10 min read
Practice Choosing a Local LLM Within a Memory Budget
That is the moment this exercise trains you for. You already know how to estimate weight storage, and you have already run one small model and watched its…
Read tutorial
intermediate
11 min read
Practice Comparing LLM Model Routes by Quality, Latency, and Cost
The cheapest route on the pricing page is often the most expensive one in production.
Read tutorial
intermediate
9 min read
Test LLM API Rate-Limit Retries and Backoff Without Live Calls
A retry loop that looks correct in code review can still double your request volume, blow past a deadline, or replay a non-idempotent call. You cannot see…
Read tutorial
intermediate
8 min read
Practice Testing LLM Cache Safety, Freshness, and Invalidation
A cache hit returns in 40 milliseconds, reads fluently, and is wrong. Nothing in the response text tells you that. Only the metadata sitting beside it can.
Read tutorial
intermediate
9 min read
Simulate LLM Batching Policies with Variable Request Arrivals
Static batch math tells you what happens when the queue is already full. Real traffic does not cooperate. Requests arrive in clumps, then nothing arrives…
Read tutorial
intermediate
10 min read
Test LLM Routing Fallbacks Under Controlled Failures
Your fallback chain looks correct in review. Then the primary stalls at 900ms, the backup is rate-limited, and nobody can say whether the request…
Read tutorial
intermediate
10 min read
Model LLM Cost, Latency, and Quality with Weighted Utility
Three engineers, three candidates, three different winners. The meeting ends the way it always does: whoever speaks last wins, and the decision gets…
Read tutorial