RAG Chunking: How to Split Documents Without Losing Meaning

The problem: chunking is a retrieval contract, not a character count

Chunking decides what the retriever is allowed to return as evidence. If a chunk is too broad, an embedding represents several unrelated claims and retrieval becomes vague. If it is too small, the answer loses the qualifier, exception, table heading, or preceding definition that gives a sentence its meaning. A fixed character limit can be a useful implementation guardrail, but it is not a semantic policy.

Start from the question the system must answer and from the evidence a reviewer would need to inspect. A support policy may need the rule, its exceptions, effective date, owner, and source locator together. A product manual may need a procedure step together with prerequisites and warnings. The right chunk is therefore a bounded evidence unit: it has one dominant purpose, enough local context to be interpreted safely, and a stable route back to the original source.

This is a narrower question than whether a company needs RAG at all. If the task can be handled by a rule, a template, a direct system lookup, or an owned human decision, use the smaller path first. The RAG systems service guide explains that architecture boundary; this article assumes retrieval is justified and focuses on making the corpus legible to it.

CHUNK-7: a controlled chunking architecture

CHUNK-7 separates document parsing from retrieval quality. It has seven stages:

1. Receive a source with a stable identifier, version, owner, access rule, and source receipt.
2. Parse the structure: titles, headings, paragraphs, lists, tables, captions, and page or section locators.
3. Bound chunks at meaningful transitions such as a heading, a procedure, a table, or an explicit exception.
4. Connect adjacent context through a small, explicit overlap or a parent-section reference.
5. Enrich every chunk with source, version, section path, locator, language, access, and lifecycle metadata.
6. Evaluate retrieval using representative questions, changed-source cases, permission-denied cases, and no-answer cases.
7. Route ambiguous, oversized, stale, or structurally damaged material to an owner instead of indexing it silently.

The model does not decide which document is authoritative. The ingestion contract does. Chunking should preserve that decision rather than erase it into anonymous text. A chunk record can be small while its provenance remains rich:

type Chunk = {
  id: string;
  sourceId: string;
  sourceVersion: string;
  sectionPath: string[];
  locator: { page?: number; heading?: string; start: number; end: number };
  text: string;
  previousChunkId?: string;
  nextChunkId?: string;
  accessRule: string;
  lifecycle: "current" | "superseded" | "quarantined";
};

The record is intentionally more than text and embedding. Without the section path and locator, a retrieved paragraph cannot be reviewed. Without lifecycle and access, an answer can look grounded while using stale or disallowed material.

Choose boundaries from document meaning

Prefer the source's own hierarchy where it exists. A heading and its first explanatory paragraph are often a more useful unit than an arbitrary 900-token window. Keep tightly coupled elements together: a rule with its exception; an instruction with the warning that limits it; a table with its title and explanatory notes; a definition with the terms it defines.

Do not force one universal size. Instead define a target range and explicit exception rules. A short definition may stand alone. A long procedure may be split by numbered steps, with each child retaining the parent procedure title. A dense table may need its rows normalized into several retrieval units, each linked to the table title, column headers, document version, and row locator. A scanned PDF whose order is unreliable should be quarantined until parsing is repaired; splitting corrupted text only makes the corruption easier to retrieve.

Overlap is a continuity mechanism, not a way to duplicate the corpus. Use it only where the next chunk depends on the preceding context: a transition sentence, a shared definition, or a heading. The overlap must remain inspectable. If the same sentence appears in many chunks, retrieval can overvalue that sentence and conceal missing coverage elsewhere.

Integration contracts: parser, index, and answer layer

The parser must output structure, not only plain text. It should identify heading depth, list order, table boundaries, captions, page breaks, and extraction confidence. The chunker then applies a versioned policy to that structure. The index receives chunks plus filters for tenant, access scope, language, document type, lifecycle, and source version. The answer layer receives selected chunks with source locators and a rule for when evidence is insufficient.

Keep deterministic checks outside the model. For example, reject a chunk if its source ID is missing, if its access rule is absent, if it begins in the middle of a heading, or if a current version has been superseded. Use the model, if needed, to suggest a semantic split for review, not to bypass the contract.

require(source.id && source.version && source.owner && source.accessRule);
require(parsed.structurePreserved && chunk.sectionPath.length);
require(chunk.text.length <= policy.maxLength || chunk.exceptionApproved);
require(chunk.lifecycle === "current");

indexable = policy.approved && evaluator.coveragePassed && !chunk.quarantined;

For a broader implementation review, case studies show how a system should be assessed through contracts, ownership, and verification rather than a feature checklist alone.

Failure modes that look like a working RAG system

Blind fixed-size windows. Retrieval finds matching words but returns a conclusion without its scope. Repair: split on structural boundaries and preserve parent context.

Detached tables and captions. A row is retrieved without the column definition or validity note. Repair: create table-aware chunks with title, headers, and locators.

Excessive overlap. Nearly identical passages dominate top-k results. Repair: keep overlap minimal and test for duplicate retrieval.

Metadata afterthought. Chunks have text but no access, version, or origin. Repair: make metadata a precondition for indexing, not a later enrichment job.

Silent source updates. A new document version is indexed alongside the old one. Repair: make lifecycle transitions explicit and test that superseded chunks cannot be retrieved.

Optimizing only for an embedding score. A benchmark looks good while human reviewers cannot locate the governing paragraph. Repair: score answer support, source precision, no-answer behavior, and correction speed alongside retrieval relevance.

Production check before rollout

Build a small evaluation set before changing the whole corpus. Include normal questions, questions needing context across a boundary, questions whose answer is a table value, denied-access questions, changed-source questions, and questions that must produce a no-answer. For each result, inspect the actual returned chunk, its parent section, version, access filter, and source locator.

Compare the candidate policy with the current baseline. Do not claim a universal “best chunk size”; the useful policy is the one that improves traceable evidence for the named corpus and task without breaking permission or lifecycle rules. Keep a rollback path: chunk-policy version, source snapshot, evaluation result, and owner for correction.

The practical outcome is not more chunks. It is a retrieval layer that can explain why this passage, from this version, under this access rule, was allowed to support an answer.