Agents Earn Trust One Tool Call at a Time
The model is not what makes an agent reliable. Its tools are, and each one earns its place by how safely it fails.
The short version
5 things that decide this
- 01An agent's reliability is the product of every tool call it makes, not a property of the model choosing them.
- 02A tool that is idempotent can be called twice by accident and still leave the world in the same state.
- 03Typed inputs stop an agent from passing a malformed argument that only fails once it reaches a real system.
- 04Reversible actions can run on the agent's own judgement. Destructive ones wait for a person to say yes.
- 05Trading CoPilot shows the split in production: it reads market alerts freely and executes trades only after a human approves.
The model is not the reliability problem
Most teams evaluating an AI agent ask which model to use. That is the wrong question to ask first. A model decides what it wants to do next. The tools decide what actually happens, and that is where an agent goes wrong in production.
An agent that can only read data cannot cause much damage, however confused it gets. An agent that can send an email, place an order or delete a row inherits the full risk of that action. A better model does not lower that risk. Reliability is not one property of the system. It is the sum of every tool call the agent is allowed to make.
That reframes the engineering work. Ask what happens when one tool call goes wrong, gets sent twice, or arrives with a bad argument. Design for that, and the model's occasional mistake stops being a production incident.
Four properties that make a tool call safe to get wrong
A tool call is safe to get wrong when getting it wrong cannot do much harm. That comes from how the tool itself is built, not from prompting the model to be careful.
Idempotent means calling the tool twice with the same input leaves the system in the same state as calling it once. Agents retry. A network call times out and the agent tries again, unsure if the first attempt landed. If the tool charges a card or sends a message on every call, a retry becomes a duplicate charge or a duplicate message. An idempotency key on the request fixes this: the tool recognizes the second call as a repeat and does nothing new.
Typed inputs catch a bad argument before it reaches a real system. A model can produce a string where a tool expects a number, or a date in the wrong format. A typed schema rejects that call at the boundary and returns a clear error. The alternative is the failure surfacing three steps downstream, where it is harder to trace back.
Reversible actions go first in what an agent is allowed to run on its own judgement. Drafting an email, tagging a record or querying a database can all be undone or cost nothing if wrong. Destructive actions, ones that spend money, delete data or send something a customer sees, sit behind a gate. The tool itself refuses to run until a human approves it, no matter how confident the model sounds.
- 01Idempotent: a repeated call, by accident or by retry, changes nothing extra.
- 02Typed: a malformed argument is rejected at the tool, not three steps downstream.
- 03Reversible-first: undoable actions run freely; destructive ones wait for a person.
- 04Gated: the gate lives in the tool's own code, not in a system prompt asking nicely.
- Read toolsQueries, lookups, alerts. Run on the agent's own judgement.
- Draft toolsReversible actions: a message queued, a record tagged.
- Approval gateA person says yes or no. The agent cannot skip this step.
- Execute toolsRuns only after approval. Idempotent, so a retry is safe.
- LogEvery call, every decision, kept for the next one.
The model proposes the same way at every stage. What changes is which tools are allowed to run without a person in the loop.
Trading CoPilot: freedom to read, a gate on every trade
We built Trading CoPilot for a forex trader who missed good setups whenever they stepped away from the screen. It watches TradingView alerts constantly, and that half of the system runs with no human check at all. Reading alerts, scoring them for confidence, filtering out noise: none of that can lose the trader money, so none of it waits for permission.
Placing a trade is different. Every alert that clears the confidence filter goes to the trader over messaging and waits for a yes or no. Only an approved alert reaches the broker connector, which executes and logs the trade. The agent never places an order on its own judgement, however clean the setup looks.
That split is the whole design. Reading a chart cannot cost the trader anything, so it runs unattended around the clock. Spending real money can, so it never runs without a person confirming it. The approval step is a property of the broker tool itself, not a rule the model was asked to remember.
Design the tool before you pick the model
Before writing a system prompt, list every tool the agent can call and sort each one by what a mistake costs. A read that returns the wrong row is an annoyance. A write that charges a customer twice is an incident. Sort first, then decide which ones can run freely and which ones need a human on the other end.
Anthropic's own guidance on building agents makes the same point about tool design. Give a model a small set of well-specified tools, not a large loosely defined one. A tool the model can misuse is a tool that eventually gets misused. Idempotency keys, strict schemas and an approval step on the risky calls apply the same idea to what happens after the model decides.
An agent earns trust the way a new employee does. It gets more freedom as the cost of a mistake goes down, and stays behind an approval step where it does not. That decision lives in the tools, one call at a time, not in a hope placed in the model.
Questions this raises
01How do you design safe tool calling for an AI agent?
Safe tool calling starts with sorting every tool by what a mistake costs. Make each one idempotent so a retry cannot duplicate its effect, and typed so a malformed argument fails at the tool instead of downstream. Let reversible actions run on the agent's own judgement, and gate destructive ones behind a human approval step built into the tool, not the prompt.
02What makes a tool call idempotent, and why does it matter for agents?
An idempotent tool call produces the same result whether it runs once or several times, usually enforced with an idempotency key on the request. Agents retry on timeouts and network errors more than a typical application. A tool that charges or messages on every call turns a routine retry into a duplicate action.
03Should an AI agent be allowed to take actions without human approval?
For reversible, low-cost actions, yes. Draft a message, tag a record, run a query. For anything destructive, anything that spends money or reaches a customer, the tool itself should require a person's approval before it runs. That holds regardless of how confident the model's output sounds.
