What an AI automation agency should hand you at the end
An AI automation agency should hand you working software plus the tests, monitoring, documentation and a named owner for failures, because the workflow is the part most likely to break after launch.
The short version
5 things that decide this
- 01The workflow is the part you see. The tests, monitoring, documentation and named owner are the parts that keep it running.
- 02Get six things in writing before you sign: acceptance criteria, tests, evals, monitoring, documentation and an owner for failures.
- 03Functional Fox shipped with 43 automated end-to-end tests in six suites, and an internal security audit found three critical defects that were fixed before handover.
- 04Where a model is involved, ask for evals: a fixed set of real cases the model is scored against before every change.
- 05The trade-off: tests and documentation show up as visible cost on the first invoice. They cost far less than the outage they prevent.
Why isn't the demo the deliverable?
A demo shows the happy path once, on clean data, with the person who built it driving. Your business runs the workflow thousands of times, on messy data, with nobody watching.
Most automation failures don't happen on day one. They happen in month three, when an API changes, a field name moves or a model update shifts an answer. If nothing catches that, you find out from a customer.
So when you compare AI automation agencies, compare what each one leaves behind. The workflow is table stakes. The rest is what you're really buying.
What should you ask an automation agency for in writing?
Six things. If an agency can't commit to them before the build, it won't produce them after.
- 01Acceptance criteria: what "done" means for each milestone, written down, so a milestone is accepted against them, not against a feeling.
- 02Automated tests: tests that run on every change and attack the rules, not only the happy path.
- 03Evals where a model is involved: a scored set of real cases the model must pass before any prompt or model change ships.
- 04Monitoring and alerts: an alert that names the broken step, sent to a person who'll act on it.
- 05Documentation: written for whoever inherits the system, including what to do when each part fails.
- 06A named owner for failures: a service agreement, or a trained handover to your own team.
What do good tests for an automation look like?
They attack the rules you care about most. A test that only checks the workflow runs is a smoke test. A useful test tries to break a rule on purpose and checks that it holds.
On Smart Site for Functional Fox, six suites hold 43 automated end-to-end tests. They cover tenant isolation, privilege escalation and sign permanence, the three things that would hurt the business most. An internal security audit before handover found three critical defects, and all three were fixed before launch.
That audit is part of what you should get, too. A list of what was checked and what was found tells you more about a build than any demo. Our AI agent production checklist lists the checks we use.
What changes when an AI model is part of the workflow?
You need evals as well as tests. A test checks code that gives the same answer every time. A model doesn't. An eval is a fixed set of real cases, with the right answers, that the model is scored against.
Run the evals before every prompt change and every model upgrade. If the score drops, the change doesn't ship. Without evals, a model update can quietly change how your workflow classifies, routes or replies, and nothing will flag it.
Who answers when an automation breaks at 2am?
Someone with a name, agreed before launch. This is the question most agencies leave vague, and it's the one that matters most once the system is live. Our post on why automations fail silently shows what happens when nobody owns it.
At Hashlogics, the system goes live with monitoring, evals where a model is involved, and documentation written for whoever inherits it. Then you choose: an ongoing agreement under a written service level, or a handover that trains your own team until they can run it without us. Under the service agreement, a month where we miss the written service levels is a month you don't pay for.
We hold ourselves to this on our own products. GreetKeeper, our AI phone receptionist, is run by the team that built it. Little Tree Confections runs its meeting-to-task n8n pipeline as a custom build plus a retainer, with error handling and execution logs built in.
Isn't all this more expensive?
On the first invoice, yes. Tests, evals and documentation are visible work, and an agency that skips them can quote lower. The cost comes back later as outages, manual checks and a rebuild when the original team is gone.
Builds at Hashlogics are fixed-price per milestone, accepted against written criteria. The tests and documentation are inside those milestones, not optional extras. See an automation that needs a babysitter is not done for the standard we hold workflows to.
How do you start?
Book a 45-minute call. If a build makes sense, an engineer writes a fixed-fee Blueprint from your real systems, with the acceptance criteria, tests and ownership plan in it. The fee is credited in full against the build. Read how we work or our business process automation service.
Questions this raises
01What should an AI automation agency deliver?+
Working software plus the parts that keep it working: written acceptance criteria, automated tests, evals where a model is involved, monitoring with alerts, documentation for whoever inherits it, and a named owner for failures after launch.
02How do you know an automation is production-ready?+
It passes tests that try to break its rules, its model passes a scored eval set, it alerts a named person when a step fails, and someone other than the builder can run it from the documentation.
03Who maintains an automation after launch?+
Agree it before you sign. Either the agency runs it under a written service level, or it trains your team in a planned handover. "We'll be around if anything comes up" is not an owner.
Related
- Business process automation →The service behind automations built to this standard.
- How we work →Audit, Blueprint, Build, Run.
- AI agent production checklist →The checks an AI system should pass before launch.
- An automation that needs a babysitter is not done →Retries, idempotent writes, dead-letter queues and named alerts.
