Prompt Versioning: Changes, Tests and Rollback
Start with the accepted business outcome
Compare templates, workflows and LLMs
Original seven-step release decision tree
Evidence, permissions, cost and recovery
$ prompt --release PROMPT-REV-7
> pin: prompt / model / schema / policy
> compare: baseline / candidate
> route: release / review / rollbackA production prompt is part of a contract among the task, permitted sources, model, validator and human owner. Changing its wording without recording the evaluation set and compatible output schema makes a behavioral change difficult to explain. This guide answers a narrow question: how to release a prompt revision with reproducible checks and a working rollback. The broader service is covered under prompt engineering; the Armenia AI specialist guide covers project selection.
Prerequisites and input data
Consider a synthetic support workflow. An assistant drafts a response using approved documents, while an operator decides whether to send it. The input contract includes locale, operator role, permitted source identifiers and output schema version. The output contains a draft, source references and a route: draft, clarify, no_answer or review.
Keep a baseline and representative fixtures before editing. Cover a normal question, missing context, conflicting sources, an inaccessible document, an instruction embedded in a retrieved document and an unsupported promise. Record the expected route and reason, not just an ideal sentence. Redact personal data or use synthetic cases.
Workflow design: the original PROMPT-REV-7 map
1. Bind task, owner and allowed routes.
2. Preserve an immutable bundle: prompt, model config, schema, source policy.
3. Bind the candidate to an evaluation-set version.
4. Run baseline and candidate against the same fixtures.
5. Check shape, evidence, access and harmful deviations separately.
6. Release to a limited traffic segment with the version ID in receipts.
7. If a gate fails, restore the accepted bundle and verify the result.A version names the entire bundle, for example support-draft/4.2, with hashes for its parts. A change to prompt text, model, schema, retrieval policy or tools requires its own release record. Record the author, reason, expected behavior, baseline comparison and rollback owner. A version number alone is not evidence.
Integrations and contracts
The application selects a pinned version for each request. A registry returns the reviewed bundle. Retrieval applies access rights before model context is assembled. An independent validator checks JSON and citations. The receipt stores the version and route without indiscriminately logging sensitive text. If the output schema changes incompatibly, update the consumer or expose a new interface. Do not silently swap versions mid-request.
{
"request_id": "synthetic-042",
"prompt_version": "support-draft/4.2",
"schema_version": "draft/2",
"source_policy": "approved-docs/3",
"input": {"locale": "en", "question": "Is there a delivery exception?"},
"output": {"route": "review", "draft": null, "source_ids": []}
}No approved source supports the exception in this synthetic example, so the request goes to a person. An absent answer is a correct outcome when the contract forbids unsupported promises. External actions need a separate AI automation path with authorization and confirmed effects.
Checks before release
Set acceptance criteria before comparing versions: required fields pass schema validation; citations exist and are visible to the actor; material claims have permitted evidence; inaccessible and conflicting data lead to no_answer or review; instructions in source documents cannot change application policy; provider failures take the defined fallback. Compare baseline and candidate on the same fixtures and review differences manually. The process owner sets thresholds using their own cases; there is no universal score.
Confirm that the previous bundle remains available and compatible with the consumer. Rehearse rollback on a test slice, including in-flight requests. Code deployment and prompt activation are separate events; the release log should link them.
Operations and improvement
After release, watch review and no_answer routes, schema failures, broken references, operator edits, latency and cost per accepted draft. These are drift signals, not standalone proof of quality. During an incident, stop promotion, point new requests back to the accepted version, identify which requests used the candidate and rerun the regression set after a fix.
Prompt versioning is controlled release of system behavior. For a bounded pilot, bring a redacted input, expected route and acceptance rule to a project discussion. Related guides: the system prompt contract and production prompt engineering.
const bundle = registry.pin(request.version);
const candidate = evaluate(bundle, fixtures);
if (!candidate.passesGates) return holdForReview();
return releaseWithRollback(bundle.id, previous.id);