How to Constrain What an Agent Can Do — Enough to Trust It in Production
"We have a human review step." This line comes up in every presales meeting, but it's worth pushing …
You must first enable the government to control the governed; and in the next place oblige it to control itself. — James Madison, The Federalist Papers, No. 51, 1788
"We have a human review step." This line comes up in every presales meeting, but it's worth pushing back with three questions: is what gets approved the action or the result? Who's the approver, and is there a record? After a rejection, does the process rerun from scratch? If those three questions go unanswered, "human review" is just a slogan.
Turning the slogan into engineering is this chapter's job — and it's also the line that separates a demo agent from a production one.
I. Start by Making the Customer's Worries Concrete
A customer's IT team doesn't worry about the agent in some vague "it's not safe" way — every worry is specific:
"Will it pass garbage parameters and blow a hole through the production system?" — if nobody validates the model-generated parameters, one misspelled table name can corrupt data.
"Will it go delete data or send emails on its own?" — an agent with delete and outbound-send permissions turns a single hallucination into a real incident.
"If it runs off the rails at 2 a.m., who notices?" — a scheduled job gets stuck in a loop at 2 a.m., and by 9 a.m. when people show up for work, it's already run a few thousand times.
"If something goes wrong, can we trace which step it was?" — a black box that just spits out a result at the end gives you nothing to localize a failure to; all you can do is start over from scratch.
Notice: what they care about isn't "is the model smart enough." They're asking whether actions have boundaries, whether the process leaves a record, whether there's a way out when something goes wrong. Four worries map to four mechanisms — the four sections of this chapter.
An agent you're willing to put into production has to pass four tests on every single action: validatable, pausable, resumable, accountable. These mechanisms already have mature, field-level designs in the open-source engineering community1; what follows here is why each one is needed and what it actually looks like.
II. Validatable: If the Parameters Don't Pass, the Action Doesn't Leave the Gate
The first gate sits at the entrance to the tool.
Every time the agent calls a tool, the parameters have to pass a structural validation: what's the tool called, which version, what types of parameters does it accept, what's the legal range for each — all written up as an explicit contract. Every call is checked against the contract first; it only executes on a match, and gets rejected with an error otherwise.
Take the smallest possible example: a "query order" tool whose contract accepts exactly two parameters — an order ID (string, fixed format) and a query reason (an enum with three options). If the model's generated call slips in an extra "customer phone number," validation fails and the action doesn't execute. This one gate alone blocks the most subtle kind of overreach — the model quietly grabbing a bit of extra data along the way.
Every tool also needs its call boundary written down explicitly: read-only, write, or outbound — pick one and state it. If a read-only tool breaks, the worst case is a bad read; if a write tool breaks, it changes state; if an outbound tool breaks, it can send data across the boundary — three completely different risk levels. Declaring this up front is what gives the tiered controls in Section V something to work from.
When self-checking on-site, ask one question: "at which step does parameter validation happen for each of your tools?" If the answer is vague, this gate is fake.
III. Pausable and Accountable: Approval Is a Ticket, Not a Pop-Up
High-risk actions have to wait for human approval — "add a human review step" is a phrase everyone knows how to say. Where's the engineering? It's in modeling approval as a ticket with fields, not a confirmation box that pops up on a screen.
A proper approval request carries at least five fields: which run it belongs to (never approve an action in isolation — you need to know what comes before and after it); which step it's stuck on (where the whole process currently stands); what it wants to do (the specific action plus parameters, at a glance); who approves it (a named person, not just "the business side"); and what the annotation says (the reasoning behind approval or rejection, kept on file).
The difference between a pop-up and a ticket is the difference between a slogan and engineering. Close the pop-up and it's gone; a ticket is a record that can be archived and audited — six months later, when the customer asks "who signed off on that outbound send," pull up the ticket: who, when, on what grounds, approved what. Four questions answered on one sheet of paper. This also sets up the handoff later: on the day the customer takes over running the system, the "who approves" field on the approval ticket is exactly where responsibility transfers (more in Chapter 17).
IV. Resumable: A Night's Delay on Approval Doesn't Mean Starting Over
Approval has one built-in property: it's slow. The approver is in a meeting, on vacation, in a different time zone — waiting overnight is completely normal.
What happens if the process can't resume? The morning after approval finally comes through, the system reruns from the beginning — the dozen-odd steps already completed, the API costs already spent, the hours already consumed, all repeated. Worse still is the behavioral distortion: a business side that can't afford to wait starts looking for ways around approval — either forcing the action to be reclassified as approval-free, or simply abandoning the process altogether. An approval nobody can wait for is the same as no approval at all.
The fix is checkpointing: after every key step, save a snapshot of the current state; once human approval clears, resume from the step where it was stuck, with all prior progress preserved exactly as it was. In open-source engineering implementations, this mechanism is called a state machine with checkpoints — every run has an ID and a lifecycle (waiting, planning, executing, waiting on a human, succeeded, failed); when it's stuck at "waiting on a human," state freezes, a snapshot lands, and it can resume at any time1.
It also happens to solve the third worry from Section I: a job running wild at 2 a.m. — the state machine has a timeout and a retry ceiling, and once a loop hits the threshold, it suspends automatically and hands off to a human. "Running a few thousand times at 2 a.m." turns into "suspended at 2 a.m., handled by a human at 9 a.m."
V. Tiered Authorization: Not Every Action Needs a Human Sign-Off
Once the four tests are in place, the easiest way to turn this into theater is the opposite failure: making every action wait for human approval. Approval is a scarce resource — approving everything is functionally the same as approving nothing (people get fatigued, and start clicking "approve" with their eyes closed).
The right answer is to tier by risk:
| Action tier | Examples | Handling |
|---|---|---|
| Read-only | Query, retrieve, summarize | Auto-execute |
| Reversible write | Write intermediate results that can be rolled back | Auto-execute + post-hoc spot check |
| Irreversible write | Write to a production table, delete, overwrite | Approval required |
| Outbound | Send an email, call a third-party API | Approval required |
The basis for tiering is exactly the call boundary each tool explicitly declared in Section II — the contract comes first, the tier comes second, one link locked into the next.
There's actually an even earlier question than tiering: should this step be handed to the agent at all? One delivery team offered a concrete way to slice this: break a workflow into roughly ten steps, and hardcode about five of them into deterministic logic — math calculations, compliance checks, the kind of thing that can't afford to be wrong, gets no room for the model to improvise and absolutely no room to make a mistake; hand three or four steps to the agent, allowing flexibility; and leave two for human review. The counterexample is just as blunt: something like financial reconciliation shouldn't be left to an agent's judgment at all — what it needs is a deterministic result, not intelligence2. The first question in risk tiering isn't "does this need approval" — it's "does this step need intelligence at all."
Now let's walk through this chapter's mechanisms end to end, using a near-disaster that got caught in time (details anonymized).
An expense-approval agent, at the "report the outcome" step, generated an outbound action: send the full quarter's expense details — employee names and bank card numbers included — as an attachment to a third-party mailbox. That mailbox came from an email disguised as a finance-system notification, which the model had treated as a legitimate configuration. What happened next, second by second: structural validation flagged it first — the outbound tool's contract stated "attachments are limited to summary reports; itemized data is prohibited," the parameters were invalid, the action never executed, and the process stopped where it was; risk tiering classified the outbound send as requiring approval, and an approval ticket was generated — which run, which step, what it wanted to send, to whom; the customer's finance lead saw the ticket and rejected it, with the annotation "that address isn't on the whitelist"; the system resumed from its checkpoint, switched to an in-app notification instead, and the eleven steps already completed stayed exactly as they were. No data ever left the boundary, and the customer's IT department saw the full record in the next day's logs. The disaster didn't fail to happen — it was stopped at the door by engineering.
VI. Four Tests, One Sentence for the Customer
Pulling the four sections together: validatable (executes only once parameters pass the contract), pausable (high-risk actions wait on a ticket), resumable (resumes from the breakpoint once approved), accountable (who approved what is on paper) (see Figure 11-1).
Figure 11-1: An action fit for production must be validatable, pausable, accountable, and resumable.
Together, these four tests are the last of the five elements of the harness a front-line company described — the supervision mechanism: low-confidence outputs go to a review queue, irreversible actions wait for human approval, and an audit trail follows every step3. Different names, same substance: turning "a human watching the agent" into "the architecture watching the agent." Front-line teams build this down to the tool layer, by permission, role, and scenario — the same tool can trigger different consequences depending on who's using it, and in what environment4.
For the customer, none of this jargon is necessary. It comes down to one sentence: every action it takes has a boundary, a record, and a stop. Every worry the customer had is now answered.
Boundaries on its actions only answer "can it do this." The moment a prompt changes, a model gets swapped, or a tool gets added, a different question shows up right away: does it still do it correctly? If that question goes unanswered after every change, all the boundaries built so far still can't hold delivery quality in place. The next chapter turns that question into a reproducible evaluation system.