LLM Response Caching: Save Cost Without Stale Answers

Caching an LLM response can avoid a repeated model call. A weak cache key or freshness rule can also replay an answer for the wrong user, source revision or task. The useful question is which exact conditions make an earlier answer valid for this request. This guide presents an inspectable architecture and pseudocode, not a universal savings claim.

For the broader design see prompt engineering and AI automation. This article focuses on reuse boundaries. Model routing decides which eligible model handles a miss; the model selection matrix helps compare accepted outcome cost. The AI specialist in Armenia page covers end-to-end engineering work.

Problem and requirements

A stable explanation of a fixed document or a repeated normalization task may be cacheable. A personal answer, live stock level, current price or revocable entitlement often is not. Caching cannot correct a wrong model answer; it makes the error faster to repeat. Begin with a written contract for task eligibility, data owner, freshness window and invalidation events. A RAG answer may become invalid when a document, index or access-control list changes.

Architecture: CACHE-GATE-7

This original seven-stage map is classify request → authorize caller → build versioned key → check freshness → validate hit → generate on miss → store and observe. Authorization precedes every cache read. An entry is bound to its tenant and permission scope. A source or policy revision changes the key or invalidates the old entry.

A key is more than raw question text. Build it from a canonical request plus tenant, permission scope, task, locale, prompt and schema versions, model policy and source revision. Hash the canonical structure when needed; do not put private data or secrets in plain keys and logs. A hash does not replace authorization.

Keys, TTL and invalidation

TTL is a backstop, not proof of correctness. If a document changes one minute after a write, a six-hour TTL can replay the wrong fact. Source revision in the key, an invalidation event and a bounded TTL work together. A revision change should route new reads to a new key immediately; old entries can be removed asynchronously within a retention limit.

A semantic cache that reuses answers for similar questions needs a separate evaluation. Embedding similarity does not prove equivalent meaning: “Can I return this item?” and “Can I return it after 30 days?” may require different decisions. Begin with exact canonical keys. Expand matching only when false-hit risk is measured for a specific task and a safe fallback is defined.

#### Minimal pseudocode

const policy = classify(request);
if (!policy.cacheable) return generateAndValidate(request);
const scope = await authorize(request.actor, request.tenant, request.sources);
const key = digest(canonicalize({ scope, task: policy.task, locale: request.locale,
  promptVersion, schemaVersion, modelPolicyVersion, sourceRevision }));
const cached = await cache.get(key);
if (cached && cached.expiresAt > now() && validate(cached, scope)) return cached.answer;
const answer = await generateAndValidate(request);
if (answer.accepted) await cache.put(key, answer, policy.ttl);
return answer;

This is a contract sketch, not production-ready code. authorize runs before reading, validate checks scope and output again, and only accepted answers are stored. For actions with an external effect, caching a draft must never replace confirmation and read-back from the destination system.

Failure modes

Cross-tenant leak. Tenant is missing from the key or rights are checked only on write. Test the same question from two customers with different documents and require separate routes.

Stale source. The TTL is live but the document revision changed. After a hit, update the source: the next request must miss or verify the new revision before serving.

Error cached as success. A provider outage, empty retrieval or unvalidated answer is stored as normal output. Keep short-lived negative entries distinct from accepted responses; do not cache an unknown outcome.

Cache stampede. Concurrent misses trigger many identical calls. Use bounded in-flight deduplication, timeouts and queue limits. If cache storage fails, keep a controlled uncached path rather than an unbounded retry loop.

False semantic match. Similar words hide a different date, role or condition. Recheck exact context and acceptance rules; often disable semantic reuse for personal or high-impact answers.

Testing and production checklist

Create fixtures for exact repeats, formatting changes, different tenants and roles, new document revisions, changed prompt/schema/model policy, expired TTL, cache failure, model failure and similar questions with different meaning. Declare the expected route for each: hit, miss, bypass, invalidate, reject or review. Inspect provenance, version and rights as well as the HTTP response.

In a pilot, measure hit rate by task class, stale-hit rate, full-path p95, model calls, cost per accepted answer and correction time. A high hit rate is not useful when wrong answers increase. Compare with an uncached control path on the same workload. Do not present hypothetical savings as a measured project result.

Before release, assign an invalidation owner, bound TTL and storage size, review encryption and retention, remove secrets from logs, and provide a cache-off switch that keeps the service usable. Prompt, schema, model, source and access-policy changes must change the key or clear relevant entries. Roll out first to a low-risk task class with a human review route. An architecture review can start from task classes, sources, permissions and accepted-answer criteria.