Control LLM Token Cost With Budgets and Routing
Measure cost per accepted answer
Budgeted routes retain validation
Original COST-GATE-6 design
Illustrative prices, no savings claim
$ cost --contract COST-GATE-6
> bind: task / budget / tariff
> route: eligible / capped
> measure: accepted answerThe price per million tokens is only one budget line. Measure the full path to an accepted answer: input, output, retries, retrieval, tools, validation and human review. This article provides a copyable estimate, not a vendor quote or a savings promise. Substitute your own prices, currency and tariff date.
For the broader project context, see AI specialist in Armenia. Related work includes prompt engineering and AI automation. This article addresses token cost control specifically; see model routing for route selection and the evaluation suite for quality tests.
What makes up the cost
Separate fixed expense, variable request cost and control cost. Input tokens include instructions, history, retrieved documents and tool results. Output tokens include the answer and structured fields. Retries, backup models and repair calls add spend. Verify each provider's current pricing rules when you calculate; tariffs can change. Keep the tariff version with the routing version.
For one task class, API = N × ((I × Pin + O × Pout) / 1_000_000) × A. N is requests per period, I/O are average input/output tokens, Pin/Pout are prices per million tokens in your chosen currency, and A is the measured multiplier for extra calls. Add retrieval, storage, tools, monitoring and human review separately. Divide by accepted answers, rather than attempts.
Copyable COST-GATE-6 estimate
This is a hypothetical monthly example in arbitrary currency units (CU), drafted October 27, 2026. Every value is an assumption, not a market price or an aicoding.am production result. Assume Pin=2, Pout=8 CU per million tokens; I=2000, O=500; extra-call multiplier A=1.15; other variable expense 0.003 CU per request; and fixed expense 50 CU monthly. Route eligibility and quality have not been evaluated.
| Requests N | API = N × 0.008 × 1.15 | Other variable | Fixed | Total | At 80% accepted: cost per answer |
|---|---|---|---|---|---|
| 1,000 | 9.20 | 3.00 | 50.00 | 62.20 | 0.07775 |
| 10,000 | 92.00 | 30.00 | 50.00 | 172.00 | 0.02150 |
| 100,000 | 920.00 | 300.00 | 50.00 | 1,270.00 | 0.015875 |
Copy the formula and replace every assumption with your own data. Calculate each task class separately if workloads differ. These scenarios illustrate how fixed expense spreads across volume; they do not forecast traffic or discounts.
Three scale scenarios
Pilot (1,000 requests). Learn the request-length distribution and acceptance rate first. Do not optimize a tariff before checking quality and access rights. Version prompts and sources for reproducibility.
Operating workflow (10,000). Group traces by task: FAQ, document search, draft and action. Set input/output caps, eligible models and retry counts per class. Watch cost per accepted answer alongside rejection rate.
High volume (100,000). Monitor daily and monthly budgets, anomalies, queue pressure and tenant-level spend. Roll out a new route gradually and roll back if lower cost produces a critical quality regression. Reforecast from actual p50/p95 token counts, not one average.
Hidden expenses and estimation errors
A shorter prompt may weaken factual retrieval and increase retries. A cheaper model may produce fewer accepted answers and more manual review. Cache keys must respect tenant, rights, source revision and expiry; a wrong hit can cost more than saved tokens. Retrieval, indexing, trace storage, alerts, tests, support and model changes all belong in TCO. Recalculate if provider pricing or caching rules change.
Do not log secrets, personal data or full user text merely for observability. Record task ID, route and prompt versions, token counts, outcome code, tariff version and rejection reason. Use limited de-identified samples for audits. A user or tenant cap prevents an unexpected bill, but needs a defined fallback: refusal, queue or human review.
How to get a sound estimate
Specify period, currency, tariff date, requests by class, input/output distributions, eligible models, retry rate and acceptance criteria. Run the same de-identified cases through the current and candidate routes; measure cost, quality, latency and human work. Model low, base and high volume and retry multipliers. Show assumptions beside all numbers, without presenting a calculation as a production invoice.
For a range estimate and TCO review, share volumes, de-identified examples, data constraints and target quality criteria. This allows a comparison without false precision.
How to control the budget
Define COST-GATE-6: (1) task class and data scope; (2) token and per-request spending caps; (3) routes eligible by evaluation; (4) bounded retries and fallback; (5) answer validation; (6) cost accounting per accepted result. A router must not bypass access policy or release an invalid answer to reduce spend.
const budget = budgetFor(task.class, tenant);
const route = eligibleRoutes(task, policy, evaluationVersion)
.filter(r => estimatedWorstCase(r, task) <= budget.perRequest)
.sort(byCostPerAcceptedAnswer)[0];
if (!route) return review('no_eligible_route');
const candidate = await callWithTokenCap(route, task, budget.tokenCap);
const verdict = validate(candidate, task);
recordUsage({ route, tokens: candidate.usage, tariffVersion, verdict });
return verdict.accepted ? candidate.answer : boundedFallbackOrReview(verdict);Track daily burn rate, monthly projection, requests near the limit, tokens by task class, cost per accepted answer, critical failures and human review. An alert should explain whether growth came from volume, context length, retries, route changes or tariff. Test a new route on comparable tasks and stop rollout on quality or policy regressions even when tokens cost less.
const route = eligibleRoutes(task, policy).filter(withinBudget)[0];
if (!route) return review("no_eligible_route");
const answer = await callWithTokenCap(route, task);
return validate(answer) ? accept(answer) : boundedFallbackOrReview(answer);