AI agent production checklist
Fourteen gates, grouped by what each one protects. An agent that clears them can be trusted with real work.
What the checklist covers
4 things that decide this
- 01An agent is ready when you can say what it may do alone, prove it still works, and name who fixes it at 2am.
- 02Two gates get skipped more than any others: a frozen evaluation set, and an idempotency key on every action that changes something.
- 03The demo proves the agent can do the task once. Production asks whether it does the task correctly the four-hundredth time, unwatched.
- 04Most of these gates are cheap before launch and expensive after, because afterwards you are changing a system people already rely on.
An agent that works, and one you can leave alone
An agent that books an appointment in a demo and one that answers a repair shop's phone all day are different pieces of engineering. The second handles a caller who changes their mind. A supplier who never picks up. A tool that times out mid-sentence.
ZhoopZhoop runs both directions of that. Inbound voice agents take customer calls and confirm bookings. Outbound agents call suppliers for parts pricing. Every transcript, call record and summary lands where a person can audit it. That build is what the checklist below had to satisfy.
Each item is a gate rather than advice, because each one has a yes or no answer. A gate you cannot answer yes to is your next piece of work.
- CorrectnessIt does the right thing on known cases
- Blast radiusIt cannot do damage outside its scope
- RecoveryA failed run can be replayed or undone
- VisibilitySomebody sees the failure before a customer
- OwnershipA named person answers the page
Skipping any group does not stop the launch. It decides which kind of incident you get.
The fourteen gates
The full checklist. Every gate is a yes or no question you can put to whoever built the agent.
| Group | The gate | Why it exists |
|---|---|---|
| Correctness | A frozen set of real cases with known-correct answers exists | Without it, nobody can prove a change made things better |
| Correctness | That set runs automatically before any change ships | A prompt fix for one case routinely breaks four others |
| Correctness | Retrieval is scored separately from the final answer | Otherwise you cannot tell which half of the system broke |
| Correctness | The model version is pinned, not floating | Providers ship new versions underneath you |
| Blast radius | Every action the agent can take is listed and approved | An agent with an open toolset has an undefined job |
| Blast radius | Actions that cost money or cannot be undone need a person | This is the boundary that has to be decided, not discovered |
| Blast radius | The agent runs with its own credentials and least privilege | It should not be able to read what its job never needs |
| Blast radius | Rate and spend limits are enforced outside the model | A loop should hit a wall, not a bill |
| Recovery | Every state-changing action carries an idempotency key | Retries are normal, and duplicates are the usual result |
| Recovery | A tool timeout fails loudly instead of being guessed around | A model asked to cover a gap will invent something plausible |
| Recovery | The previous version can be restored without a rebuild | Rollback is the only fix that works at 2am |
| Visibility | Inputs, tool calls and outputs are logged for every run | An unlogged wrong answer cannot be investigated |
| Visibility | Somebody is alerted on error rate and latency, not just downtime | Degradation arrives long before an outage does |
| Ownership | A named person and an agreed service level are in writing | Informal ownership ends when that person changes team |
The frozen eval set and the idempotency key
A frozen evaluation set is thirty to fifty real cases with answers you have agreed are correct. It takes an afternoon to build and it is the only thing that tells you whether last week's prompt change helped or quietly hurt.
The idempotency key is smaller and even easier to skip. Any action that creates a booking, a charge or a ticket needs a key that makes running it twice the same as running it once. Networks retry, agents retry, and people click twice.
Both gates are boring, which is why they lose to demo dates. Both are also the difference between an incident you can explain and one you cannot.
- 01Build the eval set from real cases you have already seen, not invented ones.
- 02Include the cases that went wrong. Those are the ones a change is most likely to break again.
- 03Give the key to the action, not the request, so a replay of the same intent stays one event.
