Practice Testing LLM Cache Safety, Freshness, and Invalidation

A cache hit returns in 40 milliseconds, reads fluently, and is wrong. Nothing in the response text tells you that. Only the metadata sitting beside it can.

You already know which caching strategies exist — that decision belongs to the earlier caching-strategy discussion. This drill is about the judgment call that happens on every single hit: is this stored answer still true, still in scope, and still safe to hand to this caller? We will build a small eligibility checker, run four synthetic cases through it, and decide what to do with each one.

Why a Cache Hit Is a Claim, Not a Fact

A hit is not proof of correctness. It is an assertion that the conditions which made the answer valid still hold. Similarity scores stay high while truth drifts; a stale answer does not get less fluent as it gets older. That is what makes this failure mode so quiet — the system looks healthy from the outside while correctness decays underneath it.

Every hit has to pass three independent gates:

• Scope — tenant, principal, entitlement. Does this caller have the right to see this answer at all?
• Freshness — TTL, source version. Is the underlying truth still the same truth?
• Personalization — does the stored payload embed user-specific values like a balance, a renewal date, or a permission flag?

A false miss costs money. A false hit costs trust. That asymmetry should drive your default posture: when in doubt, miss.

The exercise asks you to pick one of four actions for each case: serve, serve-with-caveat, bypass, or invalidate. Bypass protects this one request. Invalidate protects every future request. Keep that distinction in your head — it decides half the answers below.

Set Up the Practice Harness

The scaffold is deliberately small: a plain script, no external services, no credentials, deterministic timestamps so every run reproduces. The entry shape carries the fields that matter:

entry = {
    "tenant": "acme",
    "principal": "user-42",
    "policy_version": "p2",
    "source_version": "42",
    "prompt_version": "7",
    "created": "2026-08-13T10:00:00Z",
    "ttl_ms": 60000,
    "response": "...",
}

The eligibility function returns a decision plus a reason code, never a bare boolean. Reasons are what you debug at 2 a.m.:

from datetime import datetime, timezone

def parse(ts):
    return datetime.fromisoformat(ts.replace("Z", "+00:00"))

def eligible(stored, live):
    # Scope: reject the candidate for this caller.
    # Invalidate only if the stored entry itself is provably wrong.
    if stored["tenant"] != live["tenant"]:
        return "bypass", "scope.tenant_mismatch"
    if stored["principal"] != live["principal"]:
        return "bypass", "scope.principal_mismatch"

    age_ms = (parse(live["now"]) - parse(stored["created"])).total_seconds() * 1000
    if age_ms >= stored["ttl_ms"]:
        return "bypass", "freshness.ttl_expired"
    if stored["source_version"] != live["source_version"]:
        return "bypass", "freshness.source_drift"

    # Version drift is a policy event: the stored answer is no longer aligned.
    if stored["policy_version"] != live["policy_version"]:
        return "invalidate", "version.policy_drift"
    if stored["prompt_version"] != live["prompt_version"]:
        return "invalidate", "version.prompt_drift"

    return "serve", "ok"

Synthetic request objects mirror the entry shape, so a mismatch shows up field by field instead of as a vague "wrong answer." Expected output is one line per case: decision, reason, and the field that decided it.

Note: The order of checks is not arbitrary. Scope failures reject the candidate for this request. Freshness failures are usually per-request, so they bypass. Version drift is a policy event, so it invalidates.

Exercise 1: The Safe Hit

Identical tenant, identical principal, unexpired TTL, matching source, policy, and prompt versions, and a payload that contains no user-specific values. Walk the function: scope passes, freshness passes, version passes. Output is serve, ok.

This case is boring on purpose. It is the control in your experiment — without a known-good baseline, you cannot tell whether a later failure came from your logic or from your test data.

Exercise 2: The Stale Hit That Looks Perfect

A near-identical query hits an entry whose source document revision changed after storage. The embedding similarity is unchanged. The response text is still fluent and confident. The only signal is source_version: 42 in the entry versus 43 in the live request.

Decision: bypass and regenerate. Then decide whether the new answer is admissible before you store it.

Common mistake: Refreshing the entry's timestamp after a superficial read instead of re-verifying the source. That converts a stale entry into a permanently stale entry with a fresh-looking clock.

The debugging signal here is nasty: hit rate looks healthy while downstream corrections rise. If users keep fixing answers that arrived fast, your cache is lying to you politely.

Exercise 3: The Personalized Hit Served to the Wrong Person

A cached answer that embedded account-specific values — a balance, a renewal date, an entitlement flag — gets reused across tenants or principals. This is not a freshness bug. It is a correctness bug, and it is the kind that ends up in a support ticket.

Two fixes exist, and they are not equivalent:

• Partition the key so the entry can never cross scope. Tenant and principal become hard partitions applied before similarity search, not soft filters applied after.
• Cache only the stable explanation and fill dynamic slots at serve time. Cache "your plan renews on the anniversary date"; inject the actual date live.

Decision: bypass for this caller, and fix the lookup path. A cross-scope hit means your partition boundary leaked. The stored entry may still be perfectly valid for its rightful tenant — deleting it would throw away a good answer and hide the real bug. The real bug is that a wrong-scope candidate was ever a candidate. Fix the partition, then add a poisoned-entry check so genuinely bad responses never enter shared storage in the first place.

Warning: Treating tenant as a post-retrieval filter is the single most common personalization mistake I see. By the time you have a similarity score, the wrong-scope answer is already a candidate.

Exercise 4: The Policy Race

The entry passes freshness and scope, but its policy_version or prompt_version no longer matches the live configuration. The stored answer was correct under the old rules. It reads as reasonable today — just no longer aligned with current policy.

This is invisible from the response text alone. You cannot read your way to the answer; you have to compare versions.

Decision: invalidate on the version-change event, not on TTL expiry. Waiting for the clock means serving known-wrong answers for the remainder of the TTL window.

One boundary worth naming: event-driven invalidation protects correctness, so it must not queue behind ordinary generation traffic. If your invalidation job waits behind a backlog of recomputes, you have built a correctness delay, not a correctness guarantee.

Choosing the Action: Serve, Bypass, or Invalidate

Bypass and invalidate are different tools. Bypass protects this request. Invalidation protects every future request. When the entry itself is unsafe — wrong policy, provably poisoned payload — bypassing is not enough; you have to remove it. When the entry is fine but the caller is wrong, the entry stays and the lookup gets fixed.

And some things should not be cached at all: irreversible side effects, live operational state, and answers that depend on the current moment. If you cannot name why the entry is still true, do not serve it.

Extend the Drill: Build Your Own Near-Miss Set

The four cases above are a starting set, not a curriculum. To make this a habit:

• Write paired queries that are lexically close but differ in exactly one dimension: tenant, negation, date, jurisdiction, or output format.
• Change one validator at a time and assert a miss. One variable per test keeps the diagnosis honest.
• Add a poisoned-entry check so known-bad responses never enter shared storage in the first place.
• Track hit quality, not just hit rate. Correlate cache hits with downstream corrections and user feedback. A high hit rate with rising corrections is a warning, not a win.

Once this feels routine, apply the same eligibility function to a tool-result cache, where freshness windows are far shorter and the cost of a stale hit is often higher.

Before you serve any hit, name the field that proves the entry is still true. If you cannot name it, bypass. Then instrument your own cache with reason codes and a near-miss test set — that is the next concrete step, and it is the one that turns this drill into a system you can trust.