Human-in-the-Loop: Where Must a Person Remain in an AI Process?

Human-in-the-loop is not a person watching every model response. It is an explicit operating contract: the system knows which work may continue automatically, which events must wait for a named reviewer, what evidence that reviewer receives, and what happens when nobody answers. Without those rules, a review button is decoration rather than control.

This guide focuses on the narrow technical question: where must a person remain in an AI-enabled workflow? It supports the broader AI specialist in Armenia service page with architecture and implementation criteria. It does not claim that every process needs AI or that human review makes an unsafe process safe by itself.

The problem: uncertainty becomes an action

Models can classify, extract, summarize and propose. Risk changes when an output crosses into an action: a customer is rejected, money is released, a record is overwritten, access is granted, a complaint is closed or a public message is sent. At that boundary, confidence is only one input. The team must also consider impact, reversibility, evidence, authority and time pressure.

A useful decomposition separates five stages:

1. Observe: collect the request, source documents, identity and current system state.
2. Infer: produce a classification, draft, score or proposed action.
3. Validate: check schema, permissions, source coverage, policy and business rules.
4. Decide: approve, reject, edit, escalate or abstain.
5. Act: write to the system of record or trigger an external effect.

Human review normally belongs between validation and consequential action. It may also be required earlier when the system cannot establish identity, permission, source quality or the applicable policy. The important design choice is not “human or AI”; it is which decision rights remain human and under what conditions.

HIL-GATE: an architecture for reviewable automation

The original HIL-GATE pattern below treats review as a first-class runtime state rather than an exception hidden inside application code.

event -> context builder -> model proposal -> deterministic validation
                                         |-> low risk + valid -> bounded action
                                         |-> review trigger -> review queue
review queue -> evidence packet -> named reviewer -> approve/edit/reject/escalate
                                         |-> signed decision -> action executor
timeout -> safe fallback; incident -> stop switch; correction -> audit + evaluation

Each transition produces a stable case_id. The evidence packet should contain the original input, model output, cited sources, rule results, confidence or uncertainty signals, proposed side effect, previous system state and recovery option. A reviewer should not have to reconstruct the case across five tools.

The architecture needs nine components:

• Context builder: captures original data and current state without silently rewriting it.
• Proposal service: returns structured output and can abstain.
• Policy validator: applies deterministic permissions, thresholds and prohibited-action rules.
• Review router: selects the queue, reviewer role, priority and deadline.
• Evidence packet: presents the smallest complete set needed for a decision.
• Decision ledger: records who decided what, when, why and from which version.
• Action executor: revalidates state immediately before the side effect.
• Fallback controller: handles timeouts, unavailable dependencies and disagreement.
• Observability loop: links overrides and incidents back to evaluation cases.

This is deliberately more than a Slack approval message. The system must preserve identity, authorization, idempotency and state between proposal and execution.

Where a human gate is mandatory

Use a human gate when one or more of the following conditions are true.

Consequences are material or difficult to reverse

Financial transfers, employment decisions, medical or legal conclusions, account suspension, safety-related instructions and contractual commitments should not be executed merely because a model score crossed a threshold. The reviewer must have the authority and competence to accept the consequence. In regulated domains, a qualified professional may be required; a generic operations approver is not a substitute.

The evidence is incomplete, conflicting or not traceable

If a claim cannot be tied to an allowed source, or two systems disagree about the customer, item or amount, automation should route to review or abstain. A fluent explanation is not evidence. The review interface must display source provenance and unresolved conflicts.

Intent, permission or identity is ambiguous

A model may understand a request while the system still cannot prove that the requester is allowed to make it. Permission checks belong outside the model. Unknown identity, cross-tenant data, unusual access elevation and a new action scope require deterministic denial or human escalation.

The case is novel or outside the evaluated distribution

Low similarity to evaluated cases, a new language pattern, an unseen document type, missing fields or a changed upstream schema are useful out-of-distribution signals. They do not prove the answer is wrong, but they show that automatic performance is not established for this case.

Policy contains legitimate exceptions

Rigid rules may reject a customer who should receive a documented exception. A model should not invent the exception, and an automation should not hide it. Route the case to a person with explicit exception authority and record the rationale.

The action represents the company publicly

Drafting may be automatic; publication often should not be. Public statements, sensitive support replies, material price changes and responses to complaints require brand, legal or operational judgment. The gate may be relaxed only for narrow templates with approved variables and proven low impact.

The REVIEW-7 trigger matrix

HIL-GATE uses REVIEW-7 to decide whether the workflow may act, must wait, or must stop.

The matrix should be configured per action, not per application. Reading a draft, sending an internal suggestion and refunding a payment have different thresholds even if the same model supports all three.

Build the review contract

A human gate needs an explicit contract between product, engineering and operations.

Trigger contract. List deterministic triggers first: amount limits, restricted fields, permission changes, policy flags, missing sources, validation errors and new action types. Add model-derived uncertainty only as a secondary signal. Avoid one global confidence threshold because confidence is not calibrated equally across tasks and classes.

Evidence contract. Specify exactly what the reviewer sees: original request, structured proposal, citations, relevant history, rule failures, changed fields and predicted impact. Hide unrelated personal data. The screen should distinguish source facts, model inference and reviewer notes.

Decision contract. Define the allowed outcomes: approve, edit and approve, reject, request information, escalate or mark as incident. Free-form comments alone are hard to audit and evaluate. Capture a reason code plus optional explanation.

Authority contract. Map each action to reviewer roles and separation-of-duty rules. The person who prepared a high-risk change may not be allowed to approve it. Authorization must be checked again when the decision is submitted.

Timing contract. Set a deadline, escalation target and safe timeout behavior. “Wait forever” creates stuck workflows; “approve automatically after 30 minutes” defeats the gate. Safe timeout usually means keep the previous state, send no external action and notify the owner.

Recovery contract. Record the pre-action state, idempotency key and rollback procedure. Approval does not remove the need for recovery because the reviewer can make a mistake or the underlying state can change before execution.

Race conditions and execution safety

Review introduces time between proposal and action. During that delay, an invoice may be paid, inventory may change, a customer may withdraw consent or another operator may edit the record. Executing the old proposal without revalidation creates a time-of-check/time-of-use failure.

The executor should therefore require:

• the same case_id and an unused idempotency key;
• a signed decision from an authorized reviewer;
• matching policy, prompt and schema versions;
• a current-state fingerprint or explicit rebase;
• revalidation of permissions and prohibited actions;
• an atomic write or compensating action;
• an immutable audit event.

If current state differs, the system should not silently apply the approved patch. It should rebuild the evidence packet and ask for a new decision when the difference is material.

Failure modes the architecture must handle

Review rubber-stamping

When queues are long and screens are opaque, reviewers click approve without inspection. Measure decision latency, override patterns and repeated approvals, but do not interpret speed alone as quality. Improve the evidence packet, reduce low-value reviews and sample approved cases for quality control.

Queue overload

If every case goes to a person, the system has created manual work with extra latency. Use shadow mode to estimate review volume before launch. Split queues by risk and skill, impose workload caps, and keep a non-AI fallback for overload.

Automation bias

Reviewers may trust the model because it appears precise. Show sources before persuasive prose, avoid default-selected approval, and expose uncertainty and policy failures. Periodically include blinded control cases to verify independent judgment where appropriate.

Missing accountability

“Operations approved it” is not enough. Store the reviewer identity, role, input versions, reason code, decision timestamp, execution result and later correction. Logs must be protected against unauthorized changes and follow the retention policy.

Stale approvals and duplicate actions

An approval may arrive after a timeout or be submitted twice. Make decisions single-use, expire them explicitly, and make the action executor idempotent. Late approval should create a visible conflict, not a second side effect.

Feedback poisoning

Reviewer edits are valuable evaluation data, but they are not automatically ground truth. A rushed edit, policy exception or compromised reviewer can be wrong. Curate corrections before using them for prompts, retrieval or training.

Testing before production

Unit tests should cover every deterministic trigger and every allowed state transition. Contract tests should verify the review payload, identity propagation, version fields and action API. Integration tests should exercise approval, rejection, edit, timeout, escalation, dependency failure and duplicate submission.

The evaluation set needs representative normal cases and deliberate edge cases:

• conflicting sources and missing evidence;
• low-confidence correct answers and high-confidence wrong answers;
• cross-tenant identifiers and permission changes;
• stale state between review and execution;
• reviewer unavailable, deadline exceeded and queue overload;
• rollback success, rollback failure and partial side effects;
• Russian and English cases if both languages are operated.

Before enabling actions, run in shadow mode: generate proposals and routes, but preserve the existing process. Compare automatic routing with actual operator decisions. Then use draft mode, where the person performs the final action. Only a narrow, reversible and evaluated low-risk slice should graduate to bounded auto-execution.

Production checklist

• A named process owner defines risk tolerance and stop conditions.
• Every consequential action has an explicit human or deterministic boundary.
• Reviewer roles are enforced by the application, not stated only in documentation.
• Evidence packets separate source facts, inference and policy results.
• Decisions are structured, attributable, versioned and single-use.
• Timeouts have a safe fallback and an escalation owner.
• State and permissions are revalidated immediately before execution.
• Writes are idempotent; recovery and rollback have been tested.
• Review volume, latency, overrides, incidents and corrections are observable.
• A stop switch disables execution without destroying the review evidence.
• Evaluation cases include failure paths and both supported languages.
• The team has criteria for reducing, increasing or removing automation.

Recommended implementation sequence

Start with one workflow slice and map its decisions, evidence, actors and side effects. Apply REVIEW-7 to each action, then write the trigger, evidence, decision, authority, timing and recovery contracts. Build the ledger and executor before polishing the model prompt. Run shadow mode, measure queue volume and disagreement, fix the evidence packet, and only then permit a reversible action.

Human-in-the-loop is successful when it creates clear decision rights and recoverable execution, not when a person is inserted into every step. For a broader assessment of where AI automation fits your organization, use the AI automation service. To inspect implementation evidence, see case studies. If you need a focused architecture review, prepare one workflow, its failure cost, representative cases and the system-of-record boundary before the discussion.