RAG Metadata and Filters: Narrow Retrieval to the Right Context

A retrieval system can return a passage that is linguistically close to a question and still be wrong for the caller. It may belong to another tenant, an old policy version, a different product, an internal audience, or the wrong language. Metadata and filters are the deterministic boundary that prevents a vector search from treating every near document as eligible context.

This article focuses on that boundary: what metadata a RAG system needs, where filters must run, how failures appear, and how to test the design. It complements the broader RAG systems service guide; it does not replace a full architecture or commercial evaluation.

1. The problem: semantic similarity does not establish applicability

Vector retrieval answers a limited question: which stored passages look related to this query according to an embedding model? It cannot establish who may see a passage, whether it is current, whether it applies to a country or plan, or whether it is the authoritative version.

Consider an internal assistant for product support. A question about cancelling a subscription may retrieve a clear cancellation procedure. Without metadata, the procedure could be for an enterprise contract, a retired product, or an internal escalation team. Its text can be perfectly relevant while its use is unsafe. The failure is not fixed by raising topK, choosing a larger model, or writing a stricter prompt.

The practical rule is simple: similarity produces candidates; filters decide eligibility. The answer layer receives only candidates that are permitted, current, and in scope for the current request.

2. META-7: a compact retrieval contract

META-7 is an implementation checklist for turning documents into governed retrieval evidence:

1. Identify the source and its owner with a stable document and passage ID.
2. Classify the passage by tenant, audience or role, language, document type and product or business scope.
3. Version effective date, lifecycle, source revision and an inspectable locator.
4. Authorize the caller before content is selected for the model context.
5. Filter deterministically before ranking or immediately after a retrieval engine when its access model requires it.
6. Verify that the returned evidence is sufficient, not merely eligible.
7. Route to a cited answer, a review queue, or a clear no-answer.

The exact field names can differ, but the meaning must survive ingestion, indexing, query construction, ranking, answer generation and logging. A language field that disappears at chunking, or a tenantId that the retrieval adapter never queries, is not a control.

type RetrievalMetadata = {
  tenantId: string;
  audience: "public" | "customer" | "employee" | "admin";
  language: "en" | "ru" | "hy";
  documentType: "policy" | "manual" | "release-note";
  productId?: string;
  lifecycle: "current" | "superseded" | "quarantined";
  effectiveFrom?: string;
  sourceVersion: string;
  locator: string;
};

Metadata is not a pile of optional labels. Each field needs an owner, an allowed value set, a refresh event and a query-time purpose. If no decision uses a field, remove it. If a decision depends on a fact that has no field, add the field before trusting retrieval.

3. Architecture: apply policy before the model sees context

There are three useful stages. At ingestion, validate the source, create passage-level metadata, and reject records that lack a required owner, version or access classification. At retrieval, construct filters from the authenticated caller and the request context; these fields must come from trusted systems, not from a free-text user prompt. At answer time, verify that the selected passages actually support the answer, cite them, and retain the decision trail.

For a multi-tenant application, tenantId is not a relevance preference. It is an isolation boundary. The retrieval call should be impossible to make without it. For role-sensitive material, use an allow-list or policy decision derived from identity claims rather than asking the model to decide what a caller is allowed to read.

Language and lifecycle need similar care. A system serving Armenian, Russian and English can prefer the caller's language, but must not silently turn a language fallback into a claim that the source says the same thing. A superseded document should normally be excluded before ranking, not placed at rank ten and hoped away by the prompt.

const scope = policyScope(authenticatedCaller, request);

const candidates = vectorIndex.search(queryVector, {
  topK: 20,
  filter: {
    tenantId: scope.tenantId,
    audience: { $in: scope.allowedAudiences },
    language: { $in: scope.languages },
    lifecycle: "current",
    productId: scope.productId,
  },
});

const evidence = candidates.filter((item) => item.metadata.sourceVersion);

The example deliberately keeps authorization outside the model. A model may help interpret a query, but it should not create permissions or weaken a deterministic scope.

4. Common failure modes

Chunk metadata does not match document metadata. A document may have an access label while its chunks are indexed without it. Retrieval then cannot enforce the document policy. Copy inherited fields to each indexed unit and test a denied query.

Filters are added after context assembly. If unfiltered passages are placed into a prompt and a later component decides which ones to quote, disclosure has already occurred. Filter before context creation.

Lifecycle is descriptive only. Teams often retain old documents for audit but forget to exclude them from normal search. Use an explicit current retrieval condition and a separate audited path for historical questions.

A missing field behaves like a wildcard. Treat absent tenant, audience, language or lifecycle values as a quarantine condition, not as permission to search broadly. A permissive default converts ingestion defects into exposure risk.

Business labels are trusted from the query. A user typing “I am an administrator” should not control the role filter. Resolve scope from authenticated identity and named system-of-record facts.

Filters replace evaluation. Eligible evidence can still be incomplete, contradictory or off-topic. Filtering reduces the candidate set; it does not prove an answer. Test groundedness, citation precision and no-answer behavior separately.

5. A practical test set before production

Use a small but representative test set. Start with ordinary questions that should find current evidence. Then add a cross-tenant question, a denied-role question, a retired-version question, a mixed-language query, a product mismatch, a query with no supporting source, and a source whose metadata is incomplete. Record the expected result before tuning the system.

Log filter values, matched source IDs, source versions, locators, answer route and any policy denial. Do not log raw sensitive text merely to make observability convenient. The goal is enough evidence to reproduce a decision without recreating a second ungoverned corpus in logs.

Production acceptance gate

Before expanding a RAG pilot, assert that every retrieval request carries a trusted scope and every selected passage has inspectable provenance. Include negative tests in the release gate; a system that only demonstrates happy-path relevance has not tested its boundary.

require(scope.tenantId && scope.allowedAudiences.length > 0);
require(candidate.metadata.lifecycle === "current");
require(candidate.metadata.sourceVersion && candidate.metadata.locator);
require(candidate.metadata.language && candidate.metadata.audience);

if (!evidence.supportsClaim) route = "no-answer-or-review";
if (!policy.permits(candidate, scope)) route = "deny-without-context";

The result is not more metadata for its own sake. It is a retrieval layer that can demonstrate why a passage was eligible, which version was used, which policy constrained it, and when the system must not answer. For a bounded architecture review, start from the RAG systems guide and the practical constraints shown in the case studies.