Reranking in RAG: Why a Second Retrieval Stage Changes Answer Quality

Reranking is the narrow decision between a broad retrieval pass and the context that a model is allowed to use. It does not make a RAG system magically correct. It can make the final context more relevant when the first search stage returns a useful but noisy candidate set. The engineering task is to make that decision inspectable: which candidates entered, which model and policy version scored them, why selected passages remained eligible, and what happens when the evidence is weak.

This article addresses the long-tail architecture question. For a wider discussion of a RAG delivery, see the RAG systems service. Teams evaluating an AI engineering engagement can use the criteria here as inputs to an architecture brief, rather than treating this page as a general local-services landing page.

The problem: first-stage search optimizes recall, not final evidence

A first-stage retriever has to be fast enough to search a permitted corpus on every query. BM25, vector search, hybrid fusion or a database-native index commonly return tens or hundreds of candidates. That is appropriate for recall: a relevant passage should survive the first pass even if it is not rank one.

The answer model has a different constraint. Its context window is finite, irrelevant passages dilute instructions, and a plausible passage from the wrong source can create a confident but unsupported answer. Taking the first k candidates unchanged is therefore an uncontrolled policy. A second stage can score a query and each candidate together, then select a smaller evidence set for synthesis.

Reranking is worth evaluating only after the basics exist:

• a trusted request scope: tenant, role, locale, product and source lifecycle;
• source identity, version and a stable locator for every candidate;
• a first-stage search that has enough recall on representative questions;
• an explicit no-answer or review route when evidence is insufficient.

Without those contracts, a reranker may merely make the wrong corpus look more convincing.

A two-stage architecture: candidate retrieval, then evidence selection

The following RERANK-7 flow is a minimal system design. It is an architectural example, not a claim about a particular vendor or score threshold.

REQUEST -> trusted scope -> retrieve N eligible candidates
        -> normalize + deduplicate passage versions
        -> rerank query x candidate with pinned model/config
        -> diversity + source/evidence checks
        -> select K passages with locators
        -> answer with citations | no-answer | review

The first stage may be lexical, semantic or hybrid. Its output must retain the first-stage score and retrieval lane, but those values are not the final relevance decision. The reranker receives the normalized query, candidate text, source metadata allowed for the request, and the model/configuration version. It returns an ordering or score only for candidates that were already eligible.

Selection remains a policy decision. For example, selecting the top three results from the same outdated document can be worse than selecting two current passages from distinct sections. A production selection layer may require lifecycle current, enforce source diversity, cap duplicate chunks, and reject a high score when the required locator is missing.

Key components and their contracts

Request scope and source eligibility

Authorization belongs before retrieval and before reranking. The query cannot be a permission to search across tenants. Build a deterministic eligibility filter from authenticated identity and source metadata; apply it at first-stage retrieval; retain it through the second stage. Logging a user-visible answer is not a substitute for preventing the forbidden candidate from entering the scoring set.

Candidate record

A useful candidate record is more than plain text. It needs sourceId, immutable sourceVersion, locator, lifecycle state, locale, access attributes, first-stage rank, and a content hash. The hash lets the system reject duplicate passage versions before a reranker repeatedly promotes them. The locator makes the final citation reviewable.

Reranker configuration

Pin a model or algorithm version, preprocessing contract, requested N, selected K, and any score normalization rule. If a model is replaced, compare a representative evaluation set before silently changing answer behavior. Scores from different reranker versions are not automatically comparable; record the version with the request trace.

Evidence selector

The selector converts a ranking into allowed context. It handles deterministic rules that a probabilistic score cannot own: source lifecycle, source diversity, maximum passages per document, token budget, citation availability, and a minimum evidence condition. It should be able to return fewer than K passages or no passages at all.

Original proof: a minimal reranking contract

The following pseudocode demonstrates the boundary. It is intentionally small enough to review in a repository or test harness.

const scope = deriveTrustedScope(request.identity, request.locale);
const candidates = retrieve({ query, scope, limit: 60 });
const eligible = deduplicate(candidates).filter(isCurrentAndCitable);
const ranked = rerank({ query, candidates: eligible, model: "pinned-reranker-v7" });
const evidence = selectDiverse(ranked, { limit: 6, maxPerSource: 2, requireLocator: true });

if (!hasSufficientEvidence(evidence, query)) return routeToNoAnswerOrReview();
return answerWithCitations({ query, evidence, trace: { scope, ranked } });

This is not a claim that 60 and 6 are universal settings. They are explicit experiment parameters. A real system should evaluate candidate depth, selected depth, latency, source diversity, citation coverage and the number of safe no-answer decisions against its own task set.

Failure modes that reranking must not hide

The relevant passage never enters the candidate set

Reranking cannot promote a passage that first-stage retrieval omitted. Diagnose recall separately with exact IDs, paraphrases, multilingual cases, document updates and permission-limited queries. Increasing N may improve recall but changes cost and latency; test it instead of assuming more candidates are always safer.

Duplicate chunks dominate the context

Several overlapping chunks from one document can receive similar high scores. Deduplicate by source version and semantic span before scoring or cap them in the selector. Inspect source diversity in the evaluation output, not only aggregate relevance.

Stale or unauthorized documents receive a high score

High semantic relevance does not override access or lifecycle. Apply deterministic filters before the reranker, validate again before synthesis, and include denied-path tests. Fail closed if the filter configuration is unavailable or the candidate's source version cannot be verified.

Score changes are mistaken for quality gains

A new reranker can alter score scales while degrading the selected evidence. Compare fixed test cases with human-reviewed relevance labels and citation checks. Track version, release date, evaluation dataset revision and rollback path. Do not infer a production quality increase from a prettier score distribution.

Latency or provider failure silently changes the answer

Decide degraded behavior before an outage. A system may route to a tested first-stage-only mode, request review, or return a no-answer. That choice depends on the consequence of a poor answer. Do not silently bypass the reranker when the system's acceptance criteria require it.

Testing: measure decisions, not only model output

Start with a small, maintained evaluation set that includes normal questions, exact identifiers, paraphrases, ambiguous requests, outdated facts, denied-access queries, no-answer cases and recently changed documents. Keep a source version and expected evidence condition with every case.

Evaluate the stages separately. First-stage retrieval asks whether the needed source appears in the candidate set. Reranking asks whether the most useful eligible passages move into the selected context. The final answer asks whether it stays grounded in those passages, preserves necessary uncertainty and exposes citations. One final-answer metric cannot locate the broken boundary.

Production check before rollout

Run a bounded pilot before changing all user traffic. Save request-safe traces with scope, candidate IDs, source versions, ranks, reranker configuration, selected evidence and the final route. Redact query or document content where logs should not retain it. Use a release gate that requires both functional and safety cases.

require(scope.isTrusted && retrieval.configVersion)
require(candidate.sourceVersion && candidate.locator)
require(policy.allows(candidate, scope))
require(evalSet.exact && evalSet.paraphrase && evalSet.denied && evalSet.noAnswer)
require(release.hasRollback && trace.redactionReviewed)

if (!evidence.isSufficient) route = "no-answer-or-review"
if (reranker.failed && !degradedMode.wasEvaluated) route = "fail-closed"

The check deliberately avoids a universal score threshold. A threshold is meaningful only when tied to a domain, source set, acceptance set and error cost. The safer production claim is that a reranking change has an explicit scope, traceable evidence, evaluated failure behavior and a rollback route.

Where reranking fits in an AI engineering brief

Reranking is a component decision, not a standalone business outcome. Bring a representative query set, document types, access rules, update cadence, response consequences and existing first-stage evidence to an architecture review. The resulting design can decide whether a second stage is justified, which tests are required, and which answers must remain routed to human review.

For broader delivery evidence, visit the case studies. The next retrieval topics in this series will cover source citations and traceability; until then, keep reranking output subordinate to source lineage and visible evidence.