Practice Screening LLM Use Cases: Build a Go/No-Go Decision

You already agree with the screening criteria. Ambiguity, error cost, evidence needs, verification effort, simpler alternatives — you can recite them. Then someone hands you a messy task brief, and you freeze.

That gap is the whole problem. Screening is not a knowledge test. It is a consistency test, and most people fail it quietly: the same brief gets a confident "yes" on a sharp Monday and a nervous "maybe" on a tired Friday. The criteria did not change. Your reasoning did, and you never noticed because it stayed in your head.

This exercise drags that reasoning out where you can inspect it. You will run a small Python fixture over five synthetic task briefs, write one structured decision record per case, and execute a checker that flags missing rationales and unstated constraints. The fixture will not decide for you. It will refuse to let you leave a field blank.

If the underlying framework is new to you, the two decision guides on when not to use an LLM and how to screen a use case cover it. Here we assume you know the criteria and want to practice applying them.

What You Are Building and What You Need

The fixture is a single file. Pure standard library — no API keys, no network calls, no model access. That is deliberate: this exercise screens use cases, it does not call an LLM. If you find yourself reaching for a client library, you have misread the task.

• Python 3.10 or newer.
• One file, screen_cases.py, runnable with python screen_cases.py.
• Inputs: a list of synthetic task briefs, each carrying fields for ambiguity, error cost, evidence needs, verification effort, and simpler alternatives.
• Outputs: a disposition table plus a flag list for missing rationales or unstated constraints.

What the fixture is not: it does not know your domain, your compliance rules, or your users. It checks structure and internal consistency. Everything else is your judgment, which is exactly the point.

The Task Briefs: Five Cases That Pull in Different Directions

The cases are synthetic and generic on purpose. Each one stresses a different criterion, so the exercise teaches discrimination instead of pattern-matching.

CASES = [
    {
        "id": "C1",
        "brief": "Extract invoice totals from scanned PDFs into a ledger.",
        "ambiguity": "low",
        "error_cost": "high",
        "evidence_needs": "exact figures",
        "verification_effort": "high",
        "simpler_alternative": "OCR plus a fixed template parser",
    },
    {
        "id": "C2",
        "brief": "Route incoming support emails to one of six known queues.",
        "ambiguity": "low",
        "error_cost": "low",
        "evidence_needs": "keyword match",
        "verification_effort": "low",
        "simpler_alternative": "regex rules or a lookup table",
    },
    {
        "id": "C3",
        "brief": "Summarize long customer interviews into themes for a product team.",
        "ambiguity": "high",
        "error_cost": "low",
        "evidence_needs": "representative quotes",
        "verification_effort": "low",
        "simpler_alternative": "manual read-through",
    },
    {
        "id": "C4",
        "brief": "Draft a first-pass response to a legal discovery request.",
        "ambiguity": "high",
        "error_cost": "high",
        "evidence_needs": "exact citations",
        "verification_effort": "high",
        "simpler_alternative": "none that is safe",
    },
    {
        "id": "C5",
        "brief": "Tag product reviews by sentiment for a weekly dashboard.",
        "ambiguity": "medium",
        "error_cost": "medium",
        "evidence_needs": "label consistency",
        "verification_effort": "medium",
        "simpler_alternative": "off-the-shelf classifier",
    },
]

Look at the spread. C2 looks like a natural LLM job until you notice a lookup table wins outright. C3 is the clean LLM fit: language-variable input, low error cost, cheap verification. C4 is the honest "no automation yet" — high error cost, expensive verification, no safe simpler path. C5 is the borderline case, and it is the one that will teach you the most, because its verdict depends on a constraint the brief never states.

The Decision Record: One Structured Object per Case

For each case you write one record. The schema is small and closed, so the table stays comparable across cases.

RECORD_FIELDS = [
    "case_id",              # matches a case id
    "disposition",          # "llm" | "conventional" | "none"
    "rationale",            # why this disposition
    "constraint",           # the named condition it depends on
    "verification_plan",    # how you would check the output
    "simpler_alternative",  # what you considered first
]

Two fields carry more weight than they look.

A named constraint is mandatory because a disposition without a stated constraint is an opinion, not a decision. "Use an LLM" means nothing until you say "…provided a human reviews every output before it ships." The constraint is what makes the call falsifiable.

The simpler alternative field is mandatory even when you choose the LLM. It proves you looked. A record that jumps straight to "llm" without naming what you rejected is a record that skipped the screening.

Keep dispositions to a small closed set — llm, conventional, none — so the table reads consistently. And keep the honest escape hatch: insufficient information is a legitimate interim state, not a failure. Sometimes the right answer is "I cannot decide this yet, and here is the constraint I need."

Run It: The Checker and the Disposition Table

The checker validates structure, not correctness. It confirms every case has a record, every record has a rationale and a constraint, and every disposition comes from the allowed set.

ALLOWED = {"llm", "conventional", "none", "insufficient information"}

def check(cases, records):
    by_id = {r["case_id"]: r for r in records}
    flags, rows = [], []
    for case in cases:
        cid = case["id"]
        rec = by_id.get(cid)
        if rec is None:
            flags.append(f"{cid}: no record")
            rows.append((cid, "-", "MISSING RECORD"))
            continue
        problems = []
        if not rec.get("rationale"):
            problems.append("missing rationale")
        if not rec.get("constraint"):
            problems.append("unstated constraint")
        if rec.get("disposition") not in ALLOWED:
            problems.append("invalid disposition")
        if rec.get("disposition") == "llm" and not rec.get("simpler_alternative"):
            problems.append("no alternative considered")
        rows.append((cid, rec["disposition"], "; ".join(problems) or "ok"))
        flags.extend(f"{cid}: {p}" for p in problems)
    return rows, flags

Run it against your records and you get a table like this:

A flag means the record is incomplete or internally inconsistent — not that the disposition is wrong. C5's flag says nothing about whether an LLM is the right call. It says you did not write down why.

Notice what the checker is: a code evaluator. Deterministic, rule-based, cheap to rerun. That is the same category of verification you would want before trusting any LLM output — a fixed rule that catches structural failure without needing a human to read every row. The fixture is practicing the thing it preaches.

And keep the boundary sharp: a complete record can still be a bad judgment. Structure is necessary, not sufficient.

Read the Flags: Three Ways a Record Goes Wrong

The flags are not a grade. They are evidence about your thinking, and each pattern points at a different failure.

Missing rationale. The disposition is stated but the reasoning is absent. This usually means the call was made by feel and reverse-justified afterward. Here is a deliberately broken record:

{"case_id": "C5", "disposition": "llm", "rationale": "",
 "constraint": "dashboard refresh is weekly",
 "verification_plan": "spot-check 20 labels",
 "simpler_alternative": "off-the-shelf classifier"}

The checker emits C5: missing rationale. The fix is not to invent a reason — it is to actually decide why the LLM beats the classifier, then write that down.

Unstated constraint. The record assumes a condition the brief never supplied: a latency budget, review capacity, data sensitivity. If your disposition only holds because a human reviews every output, and the brief never promised a reviewer, you have smuggled in an assumption. Name it or drop the disposition.

Inconsistent pair. The disposition says llm while the simpler-alternative field names something that would obviously work. The record contradicts itself. When that happens, the alternative usually wins.

Change One Assumption and Watch the Table Move

Take C5, the borderline case. Right now it leans LLM because verification is cheap — you can spot-check labels against a dashboard. Now tighten one field: imagine the dashboard feeds a pricing decision, so error cost rises from medium to high, and the review step disappears.

Predict the new disposition before you rerun. My guess is you will move C5 from llm to none or insufficient information, because the cheap verification that justified the LLM is gone.

Run it and compare against your prediction. The table moved because the criteria interact: a change in one field can flip a case that looked settled. That is the general rule the experiment proves — a disposition is only as stable as the constraints it was written against.

The limit of the fixture matters here too. It cannot tell you whether your error-cost estimate was realistic. If you guessed wrong about how expensive a bad label really is, the table will happily produce a confident, well-structured, wrong answer. That still requires domain review.

What the Table Cannot Decide For You

The fixture checks completeness and internal consistency. It has no opinion on whether your ambiguity or error-cost estimates are accurate. A clean table is a starting position for a conversation, not the end of one.

High-stakes cases — anything touching money, safety, legal exposure, or people's opportunities — need a human reviewer regardless of how tidy the record looks. The record makes that review cheaper and more focused. It does not replace it.

So here is the decision rule: a complete record is the entry ticket to a go/no-go conversation, not the verdict. The fixture earns its keep by making your reasoning inspectable, comparable, and arguable.

Your next move: run the same fixture on three real tasks from your own backlog. Write the records honestly, run the checker, and fix the flags. Then hand one record to a colleague and ask them to argue against it. If they can, you learned something the table could not tell you. If they cannot, you have a decision you can defend.