MLOps and LLMOps
Know whether your model is still right today
You get graded evaluation on every change, and monitoring that surfaces drift before your users do. Plus a plan for the day your provider retires the version you built on.
The short version
4 things that decide this
- 01MLOps is the answer to one question: how would you find out your model got worse, before a customer tells you.
- 02A model can degrade without any code changing, because the world it learned from keeps moving while the model stays fixed.
- 03The evaluation set is the asset, not the model, since it is the only thing that lets any team change a prompt or swap a provider safely.
- 04Anthropic gives at least 60 days' notice before retiring a public model, which is comfortable with a test suite and frightening without one.
Nothing broke, and the answers got worse anyway
Ordinary software tells you when it fails. A model does not. It keeps returning confident, well-formed output while quietly getting a growing share of it wrong, and every log line says success.
The published evidence on this is blunt. A 2021 external validation of a widely deployed sepsis prediction model in JAMA Internal Medicine found an area under the curve of 0.63 across 38,455 hospitalisations. The model missed 67% of sepsis patients while generating alerts on 18% of all hospitalised patients.
That system was in production at scale. The failure was not a crash. It was a model behaving exactly as built, in a world that no longer matched its training data, with nobody positioned to notice.
Sourced figures
What the literature actually found
0.63
AUC for a deployed sepsis model on external validation (Wong et al., JAMA Intern Med, 2021)
67%
of sepsis patients that model failed to identify (same study)
0.090
maximum AUROC drop from temporal dataset shift on sepsis prediction (Guo et al., Sci Rep, 2022)
60
days' minimum notice Anthropic gives before retiring a public model
What we put around a model that is already live
This service assumes something is running. If nothing is built yet, start at the build pages.
A graded evaluation set written by your own experts, because a vendor grading its own output proves nothing to anyone.
That set wired into your release process, so a prompt or model change is scored automatically before it reaches a customer.
Traces kept for real runs, letting anyone replay what the system saw and what it did months after the fact.
Drift monitoring on the inputs as well as the outputs, since the data arriving usually shifts before the accuracy visibly falls.
A model migration runbook, so a retirement notice is a scheduled config change rather than an emergency.
A named owner and an escalation path, written into the contract rather than assumed.
What this is not
- A dashboard of latency and token spend. Those matter, and neither tells you whether answers are correct.
- A platform purchase. Most teams we meet need a modest evaluation habit far more than another tool.
- Grade casesYour experts write the accepted answers.
- Score changesEvery prompt or model edit, before release.
- Trace productionReal runs, replayable later.
- Watch the inputsDrift shows here first.
- Retrain or migratePlanned, against the same test set.
- Re-gradeNew failures become new cases.
The last station is what makes this a loop rather than a launch checklist. Every failure a customer finds should end up in the graded set, so the same mistake cannot ship twice.
Getting your experts to write down what good looks like
Every team we work with agrees evaluation matters. Very few have an hour of senior time to spare for grading examples, which is the only way it gets built.
So we keep it small and specific. Fifty real cases chosen for disagreement beat five hundred easy ones. We pull them from your actual traffic, sit with the people who do the job, and write down the answer they would accept. That set then belongs to you, and it is the asset that lets you fire us and keep going.
- Your experts grade, we facilitate. Never the other way around.
- Cases come from real traffic, including the ones your team argues about.
- The set is versioned, so you can see when the definition of correct changed.
- Model names live in configuration, making a provider switch a config change plus a re-run.
What we work with
Models
Pipelines
Run and watch
Systems we still run after launch
TankAware
AI + IoT petroleum site management for Sutherland Excavating Ltd.
Read the case study →
Greenlight
AI ESG and sustainability research platform.
Read the case study →
TrialTriage
AI clinical trial matching for oncology nurses and insurers.
Read the case study →
ZhoopZhoop
AI receptionist and parts procurement for a multi-branch auto repair business.
Read the case study →
“I am extremely happy with the results and would highly recommend Hashlogics to anyone.”
Daniel Khin · CEO, PremiumAudit.io
The usual monitoring setup against ours
Most AI monitoring answers whether the service responded. Ours answers whether it was right.
| Criterion | The usual approach | How we build |
|---|---|---|
| Measured on | Uptime, latency, token spend. | Correctness, against cases your experts graded. |
| Who wrote the answers | The team that built the system. | The people who do the job for a living. |
| When a prompt changes | Someone eyeballs a few outputs. | The full graded set re-runs before release. |
| Finding a wrong answer | A customer complaint, weeks later. | Input drift alerts, and the trace is replayable. |
| Model retirement | Discovered when the API errors. | Scheduled. Config change, then prove it on the test set. |
| Who owns it | Unclear after handover. | Named, with an agreed service level or a documented handover. |
01Our model is live and we suspect it has got worse. Where do we start?
Start by establishing what correct means, because without that you cannot tell degradation from a change in what users are asking. We pull a sample of recent real traffic, have your experts grade it, and score the current system against that. Whether the model got worse or the traffic changed shape, you now have a baseline that did not exist before.
02How many test cases does an evaluation set need?
Fewer than teams expect, chosen far more carefully than teams expect. Fifty cases picked because experts disagree about them will find more regressions than five hundred obvious ones. The set grows over time, because every production failure should become a case so the same mistake cannot ship twice.
03Can another model grade the answers instead of our people?
For some checks, and never as the foundation. A model judging a model can screen for format, refusals and obvious contradictions at volume. It cannot tell you whether an answer is right in your domain. Build your quality bar on it and your evaluation inherits the same blind spots as the system it tests.
04What actually causes a model to degrade?
The world moves and the model does not. Published work calls this dataset shift, and it comes from changes in technology, population and behaviour. One documented example is stark: a hospital deactivated a widely used sepsis model in April 2020 because pandemic-driven changes in patient demographics produced spurious alerting.
05Do we need a dedicated MLOps platform?
Usually not at first, and we will say so before you buy one. The habit matters more than the tooling: a graded set in version control, run in your existing CI, catches most of what a platform would. Buy the platform when the manual version is genuinely slowing you down, not before.
06What happens when our model provider retires the version we use?
It becomes a scheduled change rather than an incident, provided two things exist. The model name lives in configuration, and the graded set can prove the replacement is at least as good. Anthropic commits to at least 60 days' notice for public models, which is ample preparation time and almost none if you are starting from nothing.
07Can you work on a system another team built?
Yes, and this is where most of this work starts. Scoping conversations cost nothing. Where we have to go inside an existing codebase to answer honestly, a paid two-week diagnostic comes first. It ends in a written assessment and a fixed price.
08Is this only relevant to large ML teams?
A single AI feature in a small product needs the same question answered: how would you know it broke. The scale of the answer changes, not its necessity. Enterprise teams add access control, audit expectations and formal change approval on top of the same loop.
Related
- How to run evals on an AI agent before launch →The graded set, step by step.
- AI evals →The test suite for a model.
- Dataset shift →Why a model degrades with no code change.
- AI prototype to production →Hardening a system that has not shipped yet.
- Machine learning development →Building the model in the first place.
