Practice Auditing Agent Memory for Stale or Conflicting State

Your agent answered confidently. It was also wrong, and the memory record it trusted was three sessions old.

That is the quiet failure mode of persistent memory: nothing crashes, nothing logs an error, and the output still reads fluently. The agent simply acted on a stored claim that had gone stale, or on one of two records that contradicted each other. This exercise is the missing step between "memory exists" and "memory is trusted." You get a small set of synthetic records and a new request, and you produce a defensible disposition for every relevant claim before it influences a response.

If you already know the difference between working context, retrieved memory, and persistent state, you have the prerequisite. If not, that distinction is the thing to learn first — this exercise assumes it.

Why Stored State Goes Bad

Memory records decay in three distinct ways, and each one demands a different response.

They age out of relevance. A preference captured last week may no longer describe the user. They get contradicted by newer information. A record says the project targets one region; a later record says another. They lose their provenance. Nobody can tell who wrote the record, when, or from what source.

Here is the trap: a record can be individually plausible and still be dangerous in combination. Read one record alone and it looks fine. Read it next to a contradicting record and against the current request, and the conflict appears. That is why this is a judgment task, not a cleanup script. A script can sort by timestamp. It cannot decide whether a claim should still shape behavior.

Trusting raw memory without an audit is a real safety liability, not a theoretical one. The failure is silent because the agent still produces fluent output. The audit is not about deleting memory. It is about assigning a defensible disposition to each relevant claim before it influences a response or action.

The Four Dispositions and What Each One Means

Before you touch a record, fix your output vocabulary. Every relevant record gets exactly one of four dispositions.

Each disposition must cite the specific evidence that justifies it — the record field, the timestamp, the contradicting record. A disposition without cited evidence is a guess wearing a label.

Note: "Escalate" is not a failure of the audit. It is the audit working correctly. Some conflicts should not be resolved by the same process that found them.

What You Need Before You Start

This exercise runs on the Python standard library alone. No vector database, no API keys, no external services. The helper displays records; it does not decide anything.

You need two inputs:

1. A small synthetic dataset of memory records. Each record carries a claim text, a source or author, a timestamp, a scope (user, project, or session), and a confidence or verification status.
2. A new interaction — a request or task the agent is about to handle. This defines what counts as relevant.

Here is the sample data. Save it as records.py.

RECORDS = [
    {"id": "m1", "claim": "User prefers concise summaries.",
     "source": "user_stated", "ts": "2026-01-04", "scope": "user",
     "verified": True},
    {"id": "m2", "claim": "Project targets the EU region.",
     "source": "user_stated", "ts": "2026-01-02", "scope": "project",
     "verified": True},
    {"id": "m3", "claim": "Project targets the US region.",
     "source": "inferred", "ts": "2026-01-09", "scope": "project",
     "verified": False},
    {"id": "m4", "claim": "User is on the free plan.",
     "source": "tool_result", "ts": "2025-11-20", "scope": "user",
     "verified": True},
    {"id": "m5", "claim": "User prefers detailed technical explanations.",
     "source": None, "ts": "2026-01-06", "scope": "user",
     "verified": False},
]

NEW_INTERACTION = {
    "ts": "2026-01-10",
    "request": "Draft a project update for the EU rollout.",
}

Build the Record Inspector

The inspector does one job: surface provenance, age, and scope so your judgment has something to stand on. It computes age relative to the interaction's timestamp, not wall-clock now, so the exercise is reproducible.

from datetime import date

def parse(d):
    return date.fromisoformat(d)

def age_days(record_ts, now_ts):
    return (parse(now_ts) - parse(record_ts)).days

def inspect(records, interaction):
    now = interaction["ts"]
    for r in records:
        print(
            f"{r['id']:>3} | {age_days(r['ts'], now):>3}d | "
            f"{r['scope']:<7} | {str(r['source']):<12} | "
            f"{'verified' if r['verified'] else 'unverified':<10} | "
            f"{r['claim']}"
        )

inspect(RECORDS, NEW_INTERACTION)

Expected output:

 m1 |   6d | user    | user_stated  | verified   | User prefers concise summaries.
 m2 |   8d | project | user_stated  | verified   | Project targets the EU region.
 m3 |   1d | project | inferred     | unverified | Project targets the US region.
 m4 |  51d | user    | tool_result  | verified   | User is on the free plan.
 m5 |   4d | user    | None         | unverified | User prefers detailed technical explanations.

Confirm your setup matches before doing any judgment work. Then notice what the inspector does not do: it does not tell you which records matter. Filtering and sorting are mechanical. Deciding whether a claim should still influence behavior requires reading the claim against the current request.

Run the Audit: Work Through the Records

Start with relevance. The new interaction is about the EU rollout, so m2 and m3 are directly in scope. m1 and m5 describe how the user wants output, which affects any draft, so they are in scope too. m4 describes the user's plan, which has no bearing on a project update — out of scope, no audit attention spent.

For each relevant record, ask three questions in order: Where did this come from? Is it still fresh enough to matter? Does anything else contradict it?

m1 — Keep. Source is user_stated, timestamp is six days old, and nothing contradicts it. It can shape the draft as-is.

m2 — Escalate. This is the conflict case, and it deserves the detail. m2 says the project targets the EU region, sourced to the user, eight days old, verified. m3 says the project targets the US region, inferred rather than stated, one day old, unverified.

The newer record is also the weaker one. Recency says trust m3. Provenance says trust m2. The stakes — a project update that names the wrong region — are too high to pick a winner locally. This is exactly what escalate is for: hand it to a human or a higher-authority check. Do not let the agent silently choose.

m4 — out of scope. No disposition needed. It is not relevant to this interaction.

m5 — Update. The claim is plausible and recent, but the source field is empty. Missing provenance is not the same as bad provenance — treat it as unverified, not false. Because it would shape the tone of the draft and you cannot trace where it came from, the safe move is to narrow the claim rather than trust it wholesale. Update it to a weaker, explicitly unverified form: "User may prefer detailed technical explanations (unverified)." The updated record can inform tone only if the draft also states the uncertainty, or it can be confirmed with the user before the next interaction.

m4 revisited — Expire. Suppose the interaction had been about billing. Then m4 would be relevant, and its 51-day age plus the fact that plan status changes frequently would make it a candidate to expire and re-verify rather than trust.

Write each disposition as a short structured entry. This is the artifact the exercise produces.

m1 | keep     | source=user_stated, ts=2026-01-04, no conflict | current and sourced
m2 | escalate | conflicts with m3 (region), source=user_stated, verified | newer record is weaker, stakes high
m5 | update   | source=None, ts=2026-01-06, would shape draft tone | provenance missing, narrow claim to unverified

Where the Audit Breaks Down

The audit improves the odds. It does not guarantee correctness. Four failure modes are worth naming.

Auditing records in isolation hides context-dependent problems. A record that looks harmless alone can become harmful when combined with a specific request. This is why the audit compares records against each other and against the interaction, not just against a freshness threshold.

Missing provenance is not bad provenance. A record with no source field should be treated as unverified, not as false. Collapsing those two into one category will make you discard good information.

Freshness thresholds are a judgment call, not a universal constant. A preference from last week may be stale. A stable fact from last year may still be valid. There is no single number that works for every record type.

An audit that only filters and never escalates will silently resolve conflicts it should have handed off. If your dispositions are always keep, update, or expire, you have probably built a filter, not an audit.

Common mistake: Treating the newest record as automatically correct. Recency is one signal. Provenance and verification status are others, and they sometimes point the other way.

Modify the Exercise

One change forces you to re-examine your assumptions.

Add a sixth record with a plausible claim but no source field, and watch how the disposition changes. If you update it to an unverified form, provenance is genuinely driving your decision. If you keep it anyway, provenance is decorating the decision rather than shaping it.

Then try a second variation: give m3 a user_stated source and a verified flag. Now the newer record is also the stronger one, and the conflict resolves cleanly toward m3. Compare your dispositions before and after. The differences are the real lesson about what your audit is actually sensitive to.

A third variation: change the new interaction to a billing question. m4 moves from out of scope to relevant, and you get to test whether your scope logic holds up when the request shifts.

The Rule to Carry Forward

Before any stored claim influences an agent's response or action, it needs a disposition backed by cited evidence. Keep, update, expire, escalate — those four words are the minimum vocabulary for that decision, and the evidence citation is what separates an audit from a hunch.

The next step is to run this same pass on a real agent's memory store. Start with the records most likely to be stale or conflicting: anything with a missing source field, anything older than your interaction window, and any pair of records that describe the same entity with different values. Audit those first. The rest can wait.