RAG Evaluation Set: How to Build Questions, Answers and Sources
A RAG release needs versioned cases, expected evidence and a safe route — not a single impressive answer
Questions, source locators, reference claims, denied cases, no-answer behavior and reviewable run receipts
Original EVALSET-10 public fixture with a reproducible 12-case method and explicit limitations
RAG evaluation dataset, grounded answers, source citations, retrieval testing, safe no-answer behavior and EVALSET-10
$ evaluate-rag --contract EVALSET-10
> define: question / permitted source / expected claim / route
> run: retrieval / answer / citations / trace receipt
> judge: correctness / groundedness / refusal / escalation
> compare: versioned baseline / slice / threshold / uncertainty
> route: release / investigate / hold / human-reviewA RAG system cannot be accepted because one answer looks fluent. A useful evaluation set makes the expected behaviour inspectable: which question was asked, which sources are permitted, which claims are expected, when the system must ask for clarification, and when it must decline or hand off. That turns “it seemed better in a demo” into a repeatable engineering check.
This article presents EVALSET-10, a compact evaluation contract for a document-grounded assistant. It is not a claim about a customer deployment, a model leaderboard, or a universal threshold. The public fixture is deliberately small and synthetic so its method, limitations and expected routes can be read before a team applies the same pattern to its own authorised documents.
Start with decisions the assistant may make
An evaluation row is not merely a prompt and a preferred string. It records a decision boundary. For a policy assistant, a sound row can ask whether a current policy permits a specific answer; a safety row can require escalation; an access-control row can expect denial before retrieval. A useful first release separates these routes:
- Cited answer: the evidence is current, permitted and sufficient for a bounded claim.
- Clarification: a material identifier, role, locale, version or scope is missing.
- No-answer: the permitted corpus does not establish the claim.
- Denied: identity or policy does not allow the source or answer.
- Human review: the question is ambiguous, consequential or conflicts with evidence.
The broader delivery question belongs to RAG systems. This page is narrower: it describes how one RAG behaviour becomes a versioned, reviewable test. For a wider project-scope conversation, use /ai-specialist-armenia.
The minimum shape of an evaluation case
Each case should retain identity, provenance and the expected route. Avoid storing private document text merely to make a benchmark convenient. Store a durable source ID and an openable locator for the authorised reviewer; redact the fixture before it leaves the approved environment.
| Field | Required content | Why it matters |
|---|---|---|
case_id and version | stable ID, case-set version and owner | permits a regression to be located and reproduced |
| question and scope | user question, role, locale and allowed collection | prevents an answer from being graded outside its policy boundary |
| permitted evidence | source ID, document revision and locator | distinguishes a correct sentence from a grounded one |
| expected claims | reference facts or acceptable claim set | avoids treating stylistic similarity as correctness |
| expected route | answer, clarify, no-answer, deny or review | makes safe behaviour measurable rather than exceptional |
| review note | consequence, uncertainty and owner | keeps a human decision visible where it belongs |
Metrics are useful only after this contract exists. Ragas documents metrics such as context precision, faithfulness and answer accuracy; use them as implementation tools, not as a substitute for a case owner or business risk boundary. NIST’s AI RMF guidance likewise emphasizes documented, repeatable test, evaluation, verification and validation processes. Ragas metric documentation and the NIST AI RMF were checked on 2026-09-11.
Original proof: EVALSET-10 public fixture
The EVALSET-10 JSON fixture contains twelve synthetic cases: five normal cited answers, two changed-source cases, two no-answer cases, one access denial, one ambiguity clarification and one human-review route. It does not contain client documents, customer outputs, model scores or a vendor comparison.
Run it against a known configuration and record the exact model, prompt, embedding/index version, retrieval parameters and source snapshot. The expected result is not “12/12 answer accuracy.” The expected result is that every case exits by its declared safe route and every cited answer points only to the allowed fixture locator.
| Slice in the fixture | Cases | Expected observation | What a failure means |
|---|---|---|---|
| Current evidence | 5 | cited answer with matching source locator | retrieval, claim extraction or citation mapping needs inspection |
| Changed or stale source | 2 | current revision wins; stale claim is not repeated | lifecycle data or index deletion is incomplete |
| Missing evidence | 2 | no-answer with a usable next step | the system guessed past the corpus boundary |
| Scope and ambiguity | 3 | deny, clarify or human review | identity, policy or consequence routing is too weak |
This is a reproducible method fixture, not performance evidence. It shows the shape of a release gate. A production dataset must be expanded with representative queries, source changes, adversarial phrasing, locales, permissions, failures observed in operation and domain-owner review.
case: HR-014 / version=evalset-10-v1
scope: employee role / Armenia locale / policy collection
permitted: leave-policy@2026-08 / section=4.2
expected: cited_answer / claim="submit request before leave start"
reject: superseded leave-policy@2025-12
receipt: model + prompt + index + retrieved locators + reviewer decisionRun the test as a release receipt, not a chat experiment
Freeze the dataset and source snapshot for the run. Capture the request, returned answer, retrieved chunks or locators, citation links, route, evaluator result and any reviewer override. A run without a configuration receipt cannot tell whether a regression came from the model, the prompt, chunking, metadata filter, index, source lifecycle or the evaluator.
Compare like with like. A new model can improve average answer wording while making one policy slice less grounded. Show slice-level counts and the uncertainty created by a small sample; do not convert a tiny fixture into a company-wide accuracy claim. A sensible pilot has a named owner who reviews failures, classifies the cause and either repairs the system or updates the authorised expected answer.
The lifecycle discipline is detailed in RAG index updates: ingestion, versions and data removal. For source locators and reader-visible evidence, see RAG citations and traceability. A later metric discussion can distinguish retrieval coverage from response grounding; neither measurement authorises an unsafe action.
Failure cases should be part of the dataset
Teams often collect only questions that have clean answers. That makes the score comforting and the system fragile. Include cases where the source was revoked, two documents conflict, a role is not authorised, the question lacks a material identifier, the requested answer would be consequential, or no permitted evidence exists.
- A changed policy must test whether the former revision disappears from retrieval and whether the current locator is shown.
- A question outside the corpus must test an explicit no-answer instead of a plausible synthesis.
- A protected document request must test denial before a snippet appears in the answer or trace.
- A high-consequence request must test a review route even when its wording resembles an ordinary FAQ.
- An ambiguous question must test clarification before retrieval is treated as proof.
These cases are not “edge cases after launch.” They define the operating boundary. If a system may write records, make eligibility decisions, give legal, medical, safety or financial guidance, or affect a person materially, its action must remain in a separate authorised workflow with appropriate review.
Acceptance gate for a controlled pilot
| Check | Passing condition |
|---|---|
| Dataset provenance | every case has an owner, version, permitted source set and expected route |
| Reproducibility | the run stores model, prompt, index, source snapshot and retrieval receipt |
| Grounding | material answer claims have matching permitted, openable locators |
| Lifecycle | changed and deleted sources do not silently remain authoritative |
| Safe routing | no-answer, deny, clarification and review cases pass without a guessed answer |
| Decision record | failures, overrides and release decisions have a named owner |
Begin with one controlled use case and a small authorised corpus. Add cases from real, reviewed failure patterns rather than inflating the dataset with paraphrases. That leaves a team with evidence it can inspect, improve and re-run — a better basis for a controlled RAG pilot than a polished demo. To discuss that pilot, start with RAG systems or review the engineering proof layer in /case-studies.
require(case.id && case.version && case.permittedSources.length);
require(case.expectedRoute && case.expectedEvidenceLocators.length);
require(run.modelVersion && run.indexVersion && run.promptVersion);
require(testSet.normal && testSet.changed && testSet.denied && testSet.noAnswer);
if (answer.hasUnsupportedClaim) route = "hold-and-investigate";
if (source.isStale || scope.isUnknown) route = "clarify-or-no-answer";
if (risk.isHighConsequence || result.isAmbiguous) route = "human-review";