Practice Reviewing Provenance in a RAG Knowledge Base

A citation chip proves a chunk was retrieved. It does not prove the chunk was ever authorized or trustworthy.

Somebody forwards you a screenshot at 8:12 a.m. The answer looks confident, the source badge is clean, and the question underneath reads: can we verify this? You open the ingestion record, and the trail goes cold three fields in. That is the moment this exercise prepares you for.

You already know how the RAG pipeline moves documents from ingestion to grounded answers, and you have seen how manipulated content can poison a knowledge base. Now you sit in the reviewer's chair. Someone hands you an indexed document plus its ingestion record, and you have to decide whether it stays in the retrieval corpus, gets verified, gets quarantined, or gets escalated — and you have to defend the call with evidence.

The Three Questions a Provenance Review Actually Asks

The weak model is "provenance = the source URL." It feels sufficient because a URL is visible, copyable, and easy to paste into a ticket. It is also the reason so many reviews produce a verdict that collapses under the first follow-up question.

The stronger model splits RAG knowledge base provenance into three separate questions that can each fail independently:

Question	What it asks	What a failure looks like
Origin	Where did this document come from?	Unknown source system, unattributed upload, broken lineage
Authorization	Were we permitted to ingest it?	Missing, expired, or mismatched permission record
Content trust	Should we believe what it says?	Contradiction, staleness, near-duplicate of generated content

A legitimate internal wiki page can pass origin and fail content trust because it is three versions stale. A signed file can pass origin and fail authorization because it sits outside the policy scope of this index. A document with a perfect paper trail can still be wrong.

This is why "it has a citation" is not provenance. Retrieval traceability answers which chunk appeared in this answer. Ingestion legitimacy answers whether that chunk should have been in the corpus at all. Different questions, different failure modes, different fixes.

Note: Ingestion is a stage with its own failure modes, not a formality that happens before the real work. Treat it the way you would treat a build step: if it accepts anything, everything downstream inherits the problem.

The output of a review is one of four verdicts. They are not a severity ladder — they are different decisions:

• Retain — the record supports keeping the document in the corpus as-is.
• Verify — the gap is plausibly resolvable by checking a record or a source.
• Quarantine — remove from retrieval until the concern is resolved.
• Escalate — the decision belongs to someone with authority or evidence you lack.

Set Up the Review Workspace

This is a table-and-checklist exercise. No vector store, no credentials, no external services. You need a handful of synthetic indexed documents and their ingestion records, and you need to be willing to write down a reason for every call.

Each ingestion record should carry fields like these:

Field	Why it matters
document_id	Stable handle for the review and for later audits
source_system	Names the system of record, not "PDF uploaded by admin"
owner	Who is accountable for the content
classification	Sensitivity label inherited from the source
version_id	Lets you detect superseded content still indexed
ingested_at	Timestamp for staleness and sequencing
access_policy	Who is allowed to retrieve this
content_hash / signature status	Integrity and attestation signal

Run the review as ordered steps, every time, in the same order:

1. Read the record.
2. Check origin.
3. Check authorization.
4. Check content signals.
5. Assign a verdict with a written reason.

The expected output is one row per document: verdict plus a one-sentence justification tied to a specific piece of evidence. That evidence is usually a record field, but it can also be a documented content comparison — for example, "contradicts the authoritative runbook on the same topic." A verdict with no cited evidence is an opinion wearing a badge.

Common mistake: Reviewing the document's prose instead of its record. You are not grading the writing. You are tracing the paperwork — and, when content is the concern, the comparison against a known-good source.

Worked Case: A Clean Retain

Start with the baseline so you know what "no concerns" looks like before you judge anything ambiguous.

Document: doc_1042, an internal engineering runbook.

Record: source_system names the team's system of record. owner is a named team, not an individual who left. classification matches the source label. version_id is current. ingested_at is recent. access_policy restricts retrieval to the engineering group. The content hash matches, and the content agrees with the authoritative runbook on the same topic.

Verdict: Retain.

Justification: Origin matches a named system of record, authorization is explicit and current, and content is consistent with the authoritative source.

Notice that the retain verdict still requires a reason. A documented retain is what makes the next audit cheap — six months from now, someone can read your justification and see exactly which fields you checked. An undocumented retain just means nobody looked.

Common mistake: Retaining by default because the document is already indexed. Indexing is not approval. It is the thing you are reviewing.

Worked Case: Origin Is Fine, Authorization Is Not

Now the most common conflation in real reviews: origin and permission.

Document: doc_2117, a vendor contract summary.

Record: source_system is a legitimate, verifiable system. The document really did come from where it claims. But access_policy is blank, classification is missing, and there is no ingestion approval on file.

The instinct is to retain it because "we got it from a real system." That answers the wrong question. A real system tells you where the bytes came from. It does not tell you whether you were permitted to copy them into this index.

This is how a knowledge base quietly duplicates regulated or restricted content into a less-governed store. The source system had classification, retention rules, and access controls. The index may have none of them, and the copy now lives somewhere the compliance team does not know about.

The decision rule here is blunt: authorization gaps default to verify-or-quarantine, never retain-by-silence.

Which one? It depends on the shape of the gap.

• If the gap is a missing record — the approval probably exists, it just was not captured — verification may resolve it. Check the source system, confirm the policy, and either retain with the record attached or quarantine until it is.
• If the content is out of policy scope — it should never have been in this index regardless of paperwork — quarantine is the safer call.

Tip: Access controls on the source system do not automatically carry into the index. Confirm permissions are re-enforced at retrieval, not assumed from the origin.

Worked Case: Authorized but Content Is Suspect

A clean paper trail does not certify the content. This case teaches the content-side signals you can actually check.

Document: doc_3081, a topic summary.

Record: Origin is valid. Authorization is explicit and current. Every field is clean. And the content is still a problem: it contradicts an authoritative document on the same topic, and it reads like a near-duplicate of a generated summary already in the corpus.

Here the evidence for your verdict is not a single field. It is a content comparison: the contradiction against the authoritative document, plus the near-duplicate match. That is still auditable — you cite the two documents and the specific claim that conflicts — but it is a different kind of evidence than a missing access_policy value. Reviewers who only scan metadata will miss this case entirely.

This is where the two-tier idea earns its keep. Split the corpus conceptually:

• Tier 1 — authoritative layer: human-authored documents, official specs, primary sources. High authority, indexed separately.
• Tier 2 — enrichment layer: summaries, FAQs, generated analyses. Lower trust, tagged with an explicit source class, weighted down when Tier 1 results are available.

The risk you are hunting here is provenance debt: generated content that cites other generated content, where the chain back to a primary source has quietly snapped. If a widely-referenced summary turns out to be AI-generated and wrong, you cannot just delete it — you have to audit everything generated from it, and everything generated from those. The dependency graph is usually untracked, which means the realistic fix is rebuilding that section of the corpus from primary sources.

Three content checks a reviewer can run:

1. Does a human-authored primary source exist for this claim?
2. Is this a near-duplicate of another document already in the corpus?
3. Does it contradict an authoritative document on the same topic?

Decision rule: content doubt with valid authorization usually means verify or quarantine, not escalate. Escalation is for scope, policy, or intent questions — not for "this looks wrong."

When to Escalate Instead of Deciding

Escalation is not a failure verdict. It is the correct verdict when you lack the authority or the evidence to decide.

Escalate when:

• You suspect deliberate manipulation — the record was altered, or the ingestion path was bypassed.
• The question is policy or legal, and you are not the person who answers it.
• Two authoritative sources conflict, and you cannot determine which governs.
• The ingestion path itself shows evidence of being circumvented.

This is the same judgment as placing human review at the decision boundary: review belongs where impact and uncertainty justify it, not everywhere. A reviewer who escalates everything has produced no signal — they have just moved the queue. If every document in your sample comes back "escalate," the review is hiding weak analysis behind a safe-sounding word.

Warning: Escalation as a reflex is how a review process becomes theater. Reserve it for the cases where you genuinely cannot decide, and say why.

Common Review Mistakes

Consolidate these into a checklist and run it against every case:

• Treating a citation or a clean UI badge as proof of provenance.
• Confusing origin with authorization, or authorization with content trust.
• Retaining by default because removal feels disruptive.
• Ignoring version and staleness — an old authorized document can still be the wrong evidence.
• Writing a verdict with no cited evidence, which makes the review unauditable.
• Assuming access controls on the source system automatically carry into the index.

Each of these produces the same downstream symptom: a corpus that looks governed and is not.

Extend the Exercise

One modification deepens the skill more than re-reading the cases.

Add a contradicting document. Introduce a second document on the same topic as doc_3081 that disagrees with it. Re-run the review. Decide which one is authoritative, what happens to the other, and whether your verdict changes for the first document now that a conflict exists.

Then trace propagation. Add a generated summary that cites the suspect document. Follow the doubt outward: which documents now inherit lower trust, and how far does it spread before it reaches a primary source? This is provenance debt made visible.

Self-check: for each verdict, can you name the exact evidence that drove it — a record field or a documented content comparison? Would a colleague reach the same call from the same evidence? If the answer is no, the verdict is not finished.

Note: A clean review of these records does not guarantee the corpus is trustworthy. It guarantees that the documented concerns were handled. Those are different claims, and conflating them is its own failure mode.

The decision rule to carry into real work: origin, authorization, and content trust are three separate checks, and every verdict needs cited evidence behind it. Your next practical step is to pick a real corpus and try to name the human-authored primary source for ten documents in it. The ones you cannot name are where your provenance review should start.