Production Readiness Checklist for an LLM System
Score evidence for one workflow
Blockers stop the release
Copyable READINESS-6 checklist
Decision aid, not certification
$ review --contract READINESS-6
> bind: workflow / version / owner
> score: evidence / blockers
> route: go / pilot / holdThis working checklist supports a release decision for one defined LLM workflow. It asks what was tested, where the evidence lives, who accepts residual risk, and what happens when a check fails. It is a proposed decision aid, not a certification or a universal quality threshold. Download the editable Markdown template and keep the completed copy with your release record. For broader implementation context, see AI specialist in Armenia, prompt engineering, and AI automation.
When to use it
Complete the checklist before a bounded pilot, after changing a model, prompt, source, permission or tool, and before expanding the audience. Define one workflow and version. “Answer employee questions from permitted documents” can be checked; “our AI is ready” cannot. Record the owner, users, data boundary, allowed actions, expected outcome and review date. A different workflow needs its own decision.
The checklist is also useful when the answer is no. A missing owner, unbounded external action or untested permission boundary cannot be offset by a high average score. If a control does not apply to a draft-only assistant, explain why rather than marking it passed.
Criteria and scoring
READINESS-6 groups the checks into Scope (task, audience, data and refusal boundary), Evidence (sources and no-answer behavior), Evaluation (representative and forbidden cases with expected outcomes), Control (tool rights, approval, budgets and recovery), Operations (latency, cost, errors, accepted answers and incident ownership), and Change (versioning, rollout and rollback).
Score each item 0 for no evidence, 1 for partial evidence with an owner and deadline, 2 for reproducible evidence on this version, or N/A with an explanation. A BLOCKER scored 0 or 1 stops release regardless of the total. For other gaps, the risk owner may approve a bounded pilot with an explicit plan. The total is a gap map, not a release permit.
| Group | Check | Minimum evidence |
|---|---|---|
| Scope | Owner, audience and data boundary are defined | Workflow description and access matrix |
| Evidence | Unsupported answers can be declined | Cases with expected citation or refusal |
| Evaluation | Candidate version passed regression | Case set, rubric and retained results |
| Control | External actions are scoped and approved | Tool policy and denial test |
| Operations | Latency, budget and incident routes exist | Trace or dashboard and named owner |
| Change | Rollback is practical | Versions, procedure and rehearsal result |
How to complete the template
Download the Markdown checklist. Record system_version, model_version, prompt_version, policy_version, source_revision, owner, review_date, and audience. For each row, enter a score, evidence link, reviewer, date and action on failure. Keep links to private evidence in your internal copy; the public template does not collect data.
Trace a real request through input, retrieval, model, validation, optional tool call, response and decision receipt. Test an ordinary question, missing evidence, stale and denied documents, prompt injection, model timeout and budget exhaustion. Define the expected route before running each case. For a model change, run the same cases on both versions; the evaluation suite shows one reproducible method.
Do not turn a forecast into an observation. The workflow owner sets p95 latency, cost per accepted answer and rejection limits using their own constraints. If measurements are unavailable, record a pilot measurement plan and exposure cap. The latency guide and token cost model help select signals without asserting universal thresholds.
How to interpret the result
GO requires every blocker to score 2, an explicit owner decision for other gaps, and a version matching the tested system. PILOT means limited traffic and users, monitoring, a human route and a scheduled review. HOLD applies when a blocker remains open, the owner is unknown, an external action is unbounded, or rollback does not work. These are proposed decision rules, not a third-party certificate.
For example, good answer quality does not compensate for a missing test of access to another user's document: that is a hold. If blockers pass but p95 data is sparse, an owner might choose a bounded pilot with a stop condition. Write the reason instead of saying the system is “80% ready.”
What to do next
Assign each finding to an owner and a narrow fix. Rerun affected cases and critical regression cases, then save a new decision version. Before expanding traffic, compare accepted answers, latency, budget and review workload against thresholds chosen in advance. If a threshold fails, stop expansion or roll back, then verify the original workflow.
For an independent review, share the completed template and de-identified cases through the project brief. Do not put keys, personal data or confidential documents in a public file. The filled checklist makes the discussion about specific gaps rather than a vague readiness claim.
const blockers = checks.filter(c => c.priority === "BLOCKER" && c.score !== 2);
if (blockers.length) return "HOLD";
return owner.acceptsRemainingRisk ? "GO" : "PILOT";