RAG Source Quality: Why a Prompt Cannot Repair Bad Documents

Start with the source contract, not the prompt

Retrieval-augmented generation can make an answer traceable to permitted documents, but it cannot turn an unowned, conflicting or stale corpus into a reliable knowledge system. A stronger prompt may make an answer sound more careful. It cannot establish which document is authoritative, restore a missing version, apply access rules that were never modelled, or prove that the right evidence was retrieved.

The broad implementation question belongs on the RAG systems page. This guide focuses on a narrower production question: what must be true about source material before indexing it for a RAG system?

The SRC-6 source-quality architecture

Treat each document as an operational object, not just text to embed. The original SRC-6 model routes a source through six inspectable checks before it can contribute evidence to an answer.

source receipt → identify → classify authority → verify lifecycle
               → apply access → test retrieval → approve / quarantine

answer request → identity + scope → permitted candidate sources
               → ranked passages + version → evidence packet
               → bounded answer / no-answer / reviewer route

The index is a derived delivery layer. It is not the owner of truth. If a policy changes, the source owner changes or retires the document first; ingestion records the new state; retrieval then exposes only the permitted current version.

Problem and requirements: what “good enough” actually means

A source is useful for RAG only when the team can answer six practical questions:

1. Identity: what is this item, where did it come from and which stable ID represents it?
2. Authority: is it the governing policy, a helpful reference, a draft or an obsolete copy?
3. Lifecycle: who updates it, what version is current and when must it be reviewed or removed?
4. Access: which user, tenant, role or workflow may retrieve it?
5. Structure: can a reader distinguish the title, section, exception, effective date and scope without guessing?
6. Evaluation: do representative questions demonstrate that the intended passage is found, cited and handled safely when it is absent?

This is not a request for perfect documentation. It is a minimum contract for deciding whether the system may use a document as evidence. A small, owned collection is usually a better pilot corpus than a large drive full of exports, presentations and unnamed final versions.

Key components of a source-quality pipeline

1. Source inventory and stable receipt

Inventory each candidate before ingestion. Preserve a stable source ID, original location, content hash, owner, collection, received time and declared type. A receipt makes reprocessing and deletion possible: the team can explain what was indexed and can remove the correct derived chunks after a correction.

Avoid using the filename as the only identity. Files called policy-final.pdf, policy-final-2.pdf and new-policy.pdf are evidence of a lifecycle problem, not versions a retriever can resolve by confidence score.

2. Authority and precedence

Documents often describe the same subject with different force. A signed current policy, a procedure owned by an operating team, an explanatory FAQ and an old training deck should not be treated as interchangeable passages. Record a source class and precedence rule before retrieval.

When two permitted current sources conflict, the honest route is a visible conflict or reviewer escalation. Do not ask a model to choose the more persuasive sentence and call that a policy decision.

3. Lifecycle, freshness and deletion

Every retrievable source needs a named owner and a lifecycle event: published, superseded, expired, archived or removed. Attach an effective date where it matters, and make supersession explicit. A successful ingestion run is not proof that the corpus is fresh.

The production requirement is reversible change: a source owner can withdraw a document, the pipeline can locate its derived records and the system can show that they are no longer eligible for retrieval. Retain an audit receipt for the operation without retaining data beyond the declared policy.

4. Access before retrieval

Permissions belong before candidate retrieval, not after the model drafts an answer. Associate documents and chunks with the smallest usable access scope: tenant, role, team, project or named workflow. Resolve the caller identity and purpose before asking the index for candidates.

Post-filtering a broad candidate list is fragile: sensitive text may leak into logs, rerankers, prompts or cached context. If the access model is not ready, keep the source out of the RAG corpus until it is.

5. Readable structure and chunk boundaries

Chunking cannot recreate headings, tables, exceptions or scope that disappeared during export. Preserve the title, section hierarchy, page or source locator, version and applicable audience with each retrieval unit. Split on meaningful structural boundaries first; then check that a passage still carries the condition that changes its meaning.

For example, a sentence that states a limit without its “except when” paragraph is not a useful atomic fact. Its chunk should remain linked to the exception or route the question to the full source.

6. Retrieval evaluation and correction loop

Build a small reviewed set before a broad launch. It should include ordinary questions, changed-source questions, ambiguous terms, denied-access requests, retired documents, conflict cases and questions that must receive no answer. Review both retrieval and the final response: a fluent answer with the wrong passage is a failure even when its wording appears reasonable.

Keep a correction route that identifies whether the issue was in the source, metadata, access rule, chunk boundary, ranking, prompt or reviewer policy. Without this distinction, teams repeatedly tune prompts for defects that originate upstream.

Failure modes a prompt cannot fix

These are system failures, not stylistic failures. Prompt instructions such as “use only reliable sources” are useful guardrails after source governance exists; they are not a substitute for it.

Minimal SRC-6 acceptance gate

Use this original pseudocode as an architectural review aid. It is intentionally small: production systems will add storage, audit and failure-handling details appropriate to their context.

require(source.id && source.owner && source.version && source.authority);
require(source.lifecycle.active && source.accessRule && source.structurePreserved);
require(testSet.normal && testSet.changed && testSet.denied && testSet.noAnswer);

eligible = source.authority === "governing" || source.precedence.isDefined;
retrievable = eligible && caller.permitted(source.accessRule) && source.lifecycle.current;
release = retrievable && evaluation.beatsBaseline && corrections.haveOwner;

The gate does not promise a score. It requires the conditions that let a team investigate a miss, replace a bad source and decline an answer when the evidence is not sufficient.

Testing and production checklist

Before adding a collection, run the following checks against a representative slice rather than a polished demo:

• Receipt check: can the team map every returned passage to a stable source and version?
• Freshness check: does a superseded source disappear after the expected lifecycle event?
• Access check: do denied users receive neither a passage nor a hint about restricted content?
• Precedence check: does a conflict produce the governed result or a reviewer route instead of silent synthesis?
• Context check: does every test passage retain its scope, exceptions and locator?
• No-answer check: does the system decline unsupported questions without inventing a source?
• Correction check: can an owner repair one source and verify the derived index state afterwards?

Record the denominator for each result. “Most answers looked good” cannot distinguish a useful RAG system from a demo that excludes stale, restricted and ambiguous cases.

A practical first step

Choose one question class and a small corpus owned by one team. Make a source inventory, agree on authority and access boundaries, then create a reviewed test set before embedding the documents. Compare retrieval with the current baseline: search, a direct system query, a curated directory or a human queue. Only expand the corpus when the evidence shows that retrieval improves the defined outcome without weakening control.

For a bounded architecture review, bring sample questions, the source inventory, permissions, lifecycle rules and a small set of observed failures. That creates a concrete starting point for a RAG system, a technical audit or an evidence-led pilot. Related reading: RAG in plain language, when RAG is not needed and case studies.