What to Do About Runaway Bills and Jitter
A call cost pennies in the demo; the monthly bill hits five figures after launch. The demo ran smoot…
Hence the victories of a skillful fighter bring him neither reputation for wisdom nor credit for courage. — Sun Tzu, The Art of War, "Tactical Dispositions"
A call cost pennies in the demo; the monthly bill hits five figures after launch. The demo ran smoothly a hundred times in a row; the system starts glitching on day three in production. These two things usually get treated as two separate problems — one goes to finance, the other to ops.
They're actually one disease: the demo architecture never had a place for cost or failure in the first place. A demo environment naturally runs short chains, low concurrency, clean data, and a human watching the whole time — cost and failure never get fully exposed. Production strips all of that protection away, and the root cause finally shows itself. The fix is turning both cost and stability into explicit engineering objects — cost attributed by run, with a settable circuit breaker on the budget; stability managed by commitment, with graceful degradation and rollback built in.
I. Dissecting the Bill
The bill arrives at month's end, five figures. The project manager goes through it line by line, and not one charge is wrong — every single one is the system just "doing its job normally." Cost blowing out is never a one-off glitch. It's structural.
Cause one, context bloat. A demo runs a single round of Q&A; production runs a chain dozens of steps long — every step carries the entire preceding conversation forward, the context snowballs, and billing follows the context. The chain is an order of magnitude longer than the demo, so naturally the bill is too. The fix: split the task chain by step, carry only the necessary context at each step, and write intermediate conclusions to storage instead of stuffing them into the conversation.
Cause two, no caching. The same question gets asked twenty times over the course of a day, and every single time gets recomputed at full price. The fix: cache and reuse answers to high-frequency questions, and recompute only when something changes — what gets saved is repetition, not quality.
Cause three, model mismatch. Using the most expensive flagship model for a simple task is like using a Rolls-Royce Phantom to deliver takeout. The fix is Section III's subject: routing. For work like classification, extraction, and format conversion, a small model is faster, cheaper, and just as good.
Cause four, retry storms. A failed call retries automatically with no ceiling — a downstream service jitters for ten minutes, and the retries multiply the call volume several times over, with the most expensive minutes being exactly the minutes of the incident. The fix is in Section V: retries have to be bounded, back off, and have a circuit breaker underneath as a last resort.
These four causes make up the most common structural sources of a runaway bill, but look at them one by one and not a single one is "waste": every charge is compliant, every charge is necessary, every charge is doing real work.
Lesson one on cost: attribute first, optimize second — if you don't know where the money is going, optimization is just flailing blind.
II. Cost Attribution
The engineering form of attribution: every dollar spent gets traced back to which run, which tenant, which tool1.
Attribute by run, and the bill stops being a lump sum and becomes a map: what kind of task drove this month's bill up? Which customer's usage is climbing? Which tool's call volume is ballooning? A cost dashboard isn't a finance report — it's part of the runbook. The engineer on duty scans it every day, and an abnormal trend gets handled that same week, not left until month's end.
Above the dashboard sits the brake: a budget circuit breaker. Once a single run's spend crosses a threshold, it pauses automatically and hands off for human confirmation — not stinginess about saving money, but a fuse to stop the bleeding: a task chain that's run off the rails (the third worry from Chapter 11's list) becomes "paused, awaiting confirmation" with a circuit breaker, and "ran all night, see you at the bill" without one. The circuit breaker reuses exactly the mechanism from Chapter 11: crossing the threshold triggers an approval ticket, and once a human clears it, the run resumes from its checkpoint — the same mechanism, just one more scenario for it.
Run one set of numbers through a bill diagnosis: a $48,000 monthly bill, broken apart — $21,000 from context bloat (a thirty-step process chain carrying the full context at every step), $9,000 from no caching (high-frequency repeat questions recomputed at full price every time), $8,000 from model mismatch (extraction tasks running entirely on the flagship model), $10,000 from a retry storm (one downstream jitter amplified by unbounded retries). Each of the four gets its own treatment: split the chain by step, cache the high-frequency queries, route the tasks, cap the retries. Bring these structural sources down one by one, and next month's bill can drop to $13,000, with no loss of functionality.
The room for cost optimization is in the structure, not in the features.
III. Model Routing
The direct fix for mismatch is model routing: tier tasks by difficulty, and route each one to a model that's just good enough — classification and extraction go to a small model, complex reasoning over ambiguous clauses goes to a large one; for any critical write-back, beyond weighing capability and price, route it onto a verified, controlled execution path — validate first, write second, and wait for approval where needed2.
In real-world work, choosing a model under cost constraints is the norm, not the exception. The key to routing design is that it goes into the architecture, not patched on afterward — routing bolted on after the fact is one patch stacked on another; routing designed in from the start is a natural byproduct of task decomposition.
This lines up with Chapter 9's judgment: model orchestration (which task goes to which model) is itself one of the five harness elements from Chapter 92 — it isn't a cost-optimization trick, it's a high-priority piece of production architecture.
IV. Where Jitter Comes From, and How to Fix It
On to stability. First, get clear on where "glitching" actually comes from.
A production system's jitter comes from four sources: dependency jitter (a downstream API or model service timing out); output jitter (the same input producing two different outputs — the nature of a probabilistic system); data-source jitter (the customer's data updating, a field drifting); and load jitter (a concurrency spike — everyone in the company using it at once when the books close at month's end).
None of the four is the kind of thing you "fix once and it never happens again" — jitter is the normal state. What turns it into engineering is how you commit to it.
The standard practice in this field is to give a Service Level Objective (SLO), and the correct way to write one is to commit to three dimensions separately1:
- Availability — what fraction of requests return normally (99.9% and 99% describe two completely different systems);
- Latency — what fraction of requests return within a given time (a fast average is meaningless; what matters is the long tail);
- Quality — what fraction of outputs clear the passing bar.
The third is the easiest to leave out, and the most important: the quality commitment's standard comes directly from Chapter 12's eval set — "passing" isn't an adjective, it's a bar defined by the Regression Set. The reason the three dimensions get written separately is that their cost curves are different: every extra nine of availability multiplies redundancy cost; taming the latency tail takes architecture, not raw compute; and the quality bar is held up by the evaluation loop. Collapse all three into one line — "the system runs stably" — and none of the three has actually been committed to.
V. The Four Resilience Primitives: What to Do When Something Breaks
The SLO is the commitment; resilience is the mechanism that makes good on it. Four primitives, each aimed at one way things die:
Timeout — don't wait yourself to death. Every external call gets a time limit; once it's hit, count it as a failure and hand it to the next step. In a system with no timeouts, one stuck service can drag the entire chain down with it.
Retry — bounded, with backoff. Retries need a ceiling (to prevent the storm from Section I's cause four), and an exponential backoff between attempts (to give the downstream room to breathe) — retrying blindly and immediately is like pouring gasoline on a burning house.
Idempotency — a retry shouldn't charge twice. Sending the same operation a second time has to produce the same effect as sending it once. Retrying without idempotency is dangerous: after a timeout, you don't actually know whether the other side succeeded, and sending it again might mean two charges instead of one.
Graceful degradation — a tiered fallback plan. For minor trouble, a smaller model stands in (routing steps down); for a moderate failure, a rules engine takes over (falling back to deterministic logic); in the worst case, fall back to a human and a spreadsheet. This was already flagged in Chapter 8: only a plan that dares to write down "fall back to a human" dares to go live — a degradation path isn't a fig leaf for failure, it's a safety net designed on purpose. What the customer feels from this tiering is exactly where their sense of security comes from: when the system breaks, there's somewhere to go, and the sky doesn't fall (see Figure 14-1).
Figure 14-1: Post-launch engineering has to watch cost attribution, model routing, and recoverability during failure all at once.
VI. A One-Page Runbook
Pull everything in this chapter together into one page for the customer — the runbook: who watches which alert (Chapter 8's four questions), which page to flip to for which incident (the four failure classes, Chapter 13), when to hit the fallback button (the three degradation tiers), where the cost dashboard lives, what the circuit-breaker threshold is. Write it for whoever's on duty on the customer's side, not for the engineers — a runbook that only exists on the vendor's laptop is the same as no runbook at all (Chapter 8's discipline, restated here).
One last question about cost: after squeezing the structure and writing the commitments, the customer still thinks it's expensive — what then? Look first at a maintenance-dispatch scenario: an agent costs two thousand dollars a day, and what it's replacing is the decision of "which engineer to send out for the repair." If the cost of sending the wrong person is far higher than two thousand dollars, then the math can't just compare the API bill against zero. This scenario doesn't call for a better pitch — it calls for cost's first-principles question: compared to what? Compared to the human decision cost it's replacing, compared to the cost of getting it wrong — never compared to zero.
That closes out all eight walls of Part Three: data and permissions, integration, tool boundaries, controllable execution, provable results, trustworthy results, cost and stability, and transferable responsibility (the last wall gets its full treatment in Chapter 17).
Looking back, these eight walls all stand on one shared principle — the first design constraint on a production system isn't "smarter." It's "what happens when it breaks." A predictable small glitch beats unpredictable brilliance; the skilled fighter wins no glorious battles, and a well-run system has no firefighting stories to tell.
But laying the engineering foundation only answers "can the system stand on its own." Once it can, the real test is just beginning — if nobody uses it, everything before this was just cost. That's what Part Four takes up.