When Not to Use an LLM: A Practical Decision Guide

Most teams ask the wrong question first: "Can an LLM do this?" The better question is: "What does this task actually require?" An LLM is a skilled pattern matcher, not a knowledge database or a deterministic calculator. The real decision is rarely whether to use an LLM at all—it's which parts of your workflow should let an LLM make the call.

The Wrong First Question

"Can an LLM do this?" is almost always the wrong question because the answer is usually yes. An LLM can draft a response to a customer complaint. It can also generate a plausible-looking refund amount, invent a policy that doesn't exist, or confidently state the wrong order status.

The right question is: "What does this task actually require?"

An LLM generates text based on patterns it learned during training. It doesn't look things up in a database. It doesn't run calculations. It doesn't enforce rules. It produces text that looks like the right answer—and sometimes it is. The trouble starts when you need it to be right every time, in the same way, for a reason you can verify.

A wrong answer is cheap when you're drafting an email. It's expensive when it lands in a billing system, an access-control rule, or a medical note. Before you build anything, you need to know which kind of task you're dealing with.

The Six Questions That Decide

Run any proposed use case through these six questions before you commit to an LLM. Each one points to a role in your system, not just a yes-or-no verdict.

1. Is this genuinely ambiguous language work, or does a correct answer already exist somewhere?

If the task is "write a friendly subject line" or "summarize these meeting notes," there is no single correct answer. An LLM is a natural fit. If the task is "find the customer with account number 48291" or "calculate the tax on this order," a correct answer exists. The LLM is not the tool that finds it reliably.

2. What happens when the output is wrong?

Every LLM makes mistakes. The question is what those mistakes cost. If the output feeds a human review step, errors are an inconvenience. If the output directly triggers an action—a refund, a password reset, a content deletion—errors are a liability.

Here's the key distinction: low error tolerance usually means the LLM shouldn't make the final call. It doesn't mean the LLM can't appear anywhere in the workflow. An LLM can propose a refund amount while code validates it against policy and a human approves it.

3. Does the task need evidence the model can verify?

When you ask an LLM for "last quarter's revenue," it has no access to your company data. It will either refuse or generate a number that sounds plausible. If the task requires information from your systems, documents, or databases, the model can't verify anything.

This is an architecture decision, not a blanket ban. A bare model response is not a database lookup. But a surrounding workflow can give the model access to retrieval tools and databases—while keeping those systems authoritative. The LLM can summarize what the database returned. It just shouldn't be the source of the data itself.

4. Does the result need to be reproducible?

Ask an LLM the same question twice and you may get two different answers. That's a feature for creative work and a bug for anything that needs consistency: error messages, compliance outputs, configuration values, or anything audited. If two runs must produce the same result, a deterministic system should own that output.

5. What does the recurring cost look like?

Every LLM call charges you. Rules, search, and database queries scale to near-zero marginal cost. A feature that runs thousands of times per day can turn a cheap operation into a meaningful monthly bill. Compare the per-call, per-user, and per-month cost against a simpler tool before you commit.

6. Who maintains this system when the model changes?

Models get updated. APIs change. Vendor deprecations happen. If your feature depends on a specific model's behavior, you inherit a maintenance burden. A rule or a database query doesn't change its behavior when the vendor ships a new version.

Note: These six questions don't produce a single verdict. They tell you which component should own each part of the task. A requirement for evidence, determinism, or low error tolerance usually decides who owns the final truth or action—not whether an LLM can appear anywhere in the workflow.

When Rules and Traditional Software Win

If a task has a correct answer that can be coded, write the rule. Validation, format enforcement, and access control are deterministic jobs. An LLM adds risk without adding value.

Consider a form that must reject invalid email addresses or enforce a business rule like "orders over $500 require manager approval." Code handles this perfectly. It's fast, free per use, and never hallucinates. The same task through an LLM introduces a chance—however small—that the model accepts something it should reject or invents an exception that doesn't exist.

Common mistake: Reaching for an LLM because the task involves text. If the text follows a fixed pattern—parsing a standard format, checking a field against a list, applying a business rule—code is the right tool.

When Search and Databases Win

If the answer already exists in your data, retrieval beats generation. Search finds the answer. An LLM may invent it.

This is where the pattern-matcher mental model pays off. The model recognizes the shape of a correct response. It doesn't check whether the response is true.

The hybrid pattern works well here: use search or a database to retrieve the evidence, then let an LLM summarize or explain it. The LLM adds value on top of verified data. It should not be the source of the data itself.

Note: If you find yourself saying "the model should know this," stop. The model knows patterns, not your data. Retrieve first, generate second.

When a Smaller or Specialized Model Wins

A full LLM is overkill for many tasks. Classification, sentiment analysis, and topic tagging are often better served by a small encoder model than a large generative LLM. These models are faster, cheaper, and easier to run on modest hardware.

If the task doesn't require creative text generation, you probably don't need a generative model at all. A classifier that assigns a label to an input is a different tool for a different job.

When you do need generation, local and open-weight models can handle summarization and reasoning on a single GPU. But test your real workload before assuming a large context window is practical on limited hardware. A model with a 128K context window may run slowly or not at all on a laptop with modest memory.

My rule: start with the simplest workable solution and only escalate when it fails a real test. Build the narrow version. Run it against actual inputs. If a small model or a rule handles the task, you've saved yourself cost, latency, and maintenance.

The Hybrid Pattern: Let Code Own the Structure

The strongest systems rarely use an LLM alone. They combine an LLM for semantic judgment with deterministic code for structure, validation, and control.

The mistake is treating the LLM as the whole system instead of one component in a workflow. Let the LLM handle the parts that need judgment. Let code handle validation, format, and control flow.

A Worked Example: Order Support Assistant

Suppose you want to build a feature that answers customer questions about order status. Run it through the checklist:

• Ambiguity: Mixed. "Where is my order?" needs a factual answer. "What does my shipping delay mean?" needs explanation.
• Error tolerance: Low for the status itself. Telling a customer the wrong delivery date damages trust.
• Evidence: High. Order data lives in your database, not in the model's training data.
• Determinism: High for the status lookup. The same order must always show the same status.
• Cost: High volume if every customer query hits an LLM.
• Maintenance: Ongoing if the workflow depends on a specific model's behavior.

Now decompose the feature:

1. Database query fetches the actual order status. This is deterministic, verifiable, and free per use.
2. Code checks whether the order is delayed, refundable, or needs human review.
3. LLM drafts a customer-friendly explanation using the retrieved status as context.
4. Code validates that the LLM's response doesn't contradict the retrieved facts.
5. Human review catches edge cases the automated checks miss.

The LLM appears in the workflow, but it doesn't own the source of truth or the final action. It proposes an explanation. The database and code enforce reality.

Traceability Matters

Consider a workflow that extracts rules from messy documents. An LLM reads the documents and proposes structured rules. But code validates the output against a schema, rejects what fails, and flags anything that needs human review.

Add reference IDs that point each output back to a source. When something fails, you can ask "does this referenced source exist?" and "is the quoted text actually in that source?" instead of staring at an opaque output and wondering where it came from.

The system becomes more reliable not because the model is perfect, but because the workflow makes errors visible and bounded.

Tip: Design for failure. Ask what happens when the model produces a bad output, then build the check that catches it. A hybrid system is reliable because its error paths are visible, not because its model never errs.

Turning the Checklist into a Decision

Run your next feature idea through these steps before you build:

1. Ambiguity: Is there a correct answer, or is this open-ended language work?
2. Error tolerance: What breaks when the output is wrong—and who should make the final call?
3. Evidence: Does the task need facts the model can't verify?
4. Determinism: Must the same input produce the same output?
5. Cost: What does this cost per call, per user, per month?
6. Maintenance: Who owns this when the model or API changes?

Then assign each part of the task to the right component:

• If the answer already exists in your data, retrieve it before you generate it.
• If the task has a correct answer that code can enforce, write the rule.
• If the task needs creative language work, let an LLM propose—but keep validation and final actions in code.
• If a smaller model handles the job, don't pay for a larger one.

When in doubt, build the narrow version and test it against real inputs before committing. The question was never "can an LLM do this?" It was "what does this task actually require?"—and sometimes the smartest engineering decision is choosing not to use an LLM at all.

Take one feature or workflow you're currently considering and run it through the checklist. You'll likely find that some parts need an LLM, some need code, and some need both. Knowing which is which is the skill that separates demos from systems that survive real use.