RAG Index Updates: Ingestion, Versions and Data Removal

A RAG index is not a static database built once for a demo. It is a projection of source documents at a specific point in time. When a policy changes, a source is corrected, access is revoked or a document must be removed, the retrieval layer has to converge to that new source state without silently mixing old and new evidence.

This guide answers the narrow technical question of RAG index updates: ingestion contracts, versions, deletion, failure modes and production tests. For the broader product and delivery scope, see RAG systems. Teams that need an architecture review can start with an AI engineering brief; this article supplies implementation criteria rather than replacing that commercial page.

Start with the source-of-truth contract

An index update should begin with an authoritative source event, not a changed embedding alone. A source record needs a stable sourceId, tenant or access scope, origin, lifecycle, content version, hash, timestamps and a human-openable locator. Each derived chunk needs its own deterministic identifier and must retain the source version and access attributes that made it eligible.

The useful question is not “did the vector store accept an upsert?” It is “can a query observe only the intended current, permitted version of this source?” That turns ingestion into a reconciliation problem: the source registry declares desired state, workers make the index converge, and a verifier proves the result.

SOURCE CHANGE
  -> validate source and access attributes
  -> assign immutable sourceVersion + content hash
  -> create idempotent ingestion job
  -> extract / normalize / chunk with pinned configuration
  -> write versioned candidates and retrieval filters
  -> verify count, version, access and citations
  -> promote current | retry | quarantine | remove

Avoid using file name, upload time or vector-store record order as identity. Those values change too easily and make retries or a partial rollback ambiguous. A deterministic chunk key such as sourceId:version:chunkOrdinal:chunkerVersion provides a practical recovery boundary.

A versioned ingestion architecture

The INDEX-UPDATE-8 sketch below keeps the ownership boundaries visible. It is an architecture pattern, not a vendor prescription.

Separate the immutable source version from the derived-index version. The document content may be unchanged while extraction, OCR, parser, chunking or embedding configuration changes. Capturing both lets a team rebuild deliberately and explain why two retrieval runs differ.

Ingestion must be idempotent and observable

Events are often duplicated, delayed or delivered out of order. A retry-safe pipeline gives each request an event ID and each target state a version. Repeating the same event should produce the same candidate set, not duplicate chunks. A newer source version must not be overwritten by a late worker for an older one.

async function ingest(event: SourceChanged) {
  const source = await registry.get(event.sourceId);
  assert(source.version === event.version);
  assert(policy.allowsIngestion(source.scope));

  const job = await jobs.claim({ eventId: event.id, sourceId: source.id, version: source.version });
  if (job.alreadyCompleted) return job.receipt;

  const artifact = await extractNormalizeAndChunk(source, pinnedConfig);
  await stagedIndex.upsert(artifact.chunks, { sourceId: source.id, version: source.version });
  await verifyCandidateSet(source, artifact);
  await registry.promoteCurrent(source.id, source.version, artifact.receipt);
}

The pseudocode intentionally does not hide the critical decisions: access metadata enters before indexing; promotion follows verification; and the registry, rather than a worker's memory, determines current state. If the worker cannot establish that state, it should retry or quarantine instead of publishing an uncertain index.

Updates, replacements and removal are distinct operations

An update normally creates a new version and then promotes it. A replacement changes the retrieval target only after the candidate set is complete. A removal changes eligibility: the source must no longer appear in vector retrieval, keyword retrieval, reranking inputs, answer citations, caches or diagnostics that expose content.

Use a tombstone or removal registry when physical deletion is asynchronous. The tombstone is an immediately enforceable retrieval filter; downstream workers then erase or compact derived artifacts. Record the request, owner, scope, observed completion time and exceptions. Do not call a source removed merely because one collection was updated.

Deletion has limits outside the index. Backups, audit logs, legal retention, observability and vendor-managed replicas may follow separate retention obligations. Model the boundaries explicitly and involve the data owner for applicable policy; this is an engineering pattern, not legal advice.

Failure modes to design for

Partial update becomes current. Some chunks are written while extraction fails on page 48. Promotion must require expected chunk count, content hash, retrieval probes and source-version evidence, not just a successful job exit.

Late event overwrites a new version. A retry for version 7 completes after version 8. Compare-and-set the source version at promotion, and discard or quarantine stale work.

Deletion misses a secondary path. The vector index is cleaned but a lexical index, cache, reranker or citation lookup keeps the old passage. Maintain one removal inventory and test every retrieval path with a distinctive fixture.

Metadata drifts from content. A document moves teams but inherited chunk permissions remain old. Treat access scope and lifecycle as versioned ingestion inputs; reprocess or block the affected source when they change.

A scan sees an incomplete upload. The collector indexes a file while it is still being written. Use a stable-object signal, checksum or source-side finalization event before extraction.

Metrics hide the dangerous case. Average ingestion latency looks healthy while stale sources persist. Measure backlog age, promotion lag, current-version coverage, removal lag and failed verification separately.

Build a production acceptance set

The acceptance set should use safe fixture documents with unique phrases, known source IDs and distinct permissions. Run it through the same event, worker and index paths that production uses.

Test both a happy-path update and the failures that operators actually need to recover from: timeout after write, a duplicate webhook, a malformed file, a re-run with new chunking, access revocation and deletion during a worker retry. Keep a small evidence record for each run so a production incident can be reconstructed without guessing from transient logs.

A controlled path from prototype to production

Start with one source family and a bounded corpus. Name the source owner, ingestion owner and on-call decision path. Pin the parser, chunker and embedding configuration. Set an explicit maximum promotion lag and a safe behavior when that lag is exceeded: retain the prior verified version where policy allows, or return a transparent no-answer.

Then add a reconciliation scan. Event delivery alone cannot prove completeness: scheduled comparison between the source registry and index receipt catches missing events, orphaned chunks and failed removals. The related guides on RAG metadata and filters, RAG access control and RAG citations cover the neighboring contracts that make the index trustworthy at query time.

Before widening the corpus, review the failure budget: what happens when the index is behind, a source is disputed, a deletion request arrives, or verification cannot run? A reliable RAG update path is not the fastest path that writes vectors. It is the path that can show which version is searchable, who could see it, why it is current and how to stop serving it safely.