Self-hosted vs managed LLM
The argument is usually framed as cost. It is really about who is awake when the model stops responding at two in the morning.
The short answer
Use a managed API unless a rule or a contract says the data may not leave your network, because self-hosting replaces a bill with a team.
Teams reach for self-hosting to save money, then discover the model was the cheap part. Someone has to size the hardware, keep it patched, handle traffic spikes, and answer for quality when a new version behaves differently.
That work is worth doing when there is a real constraint behind it. Data that cannot leave a network is a real constraint. A preference for owning things is not.
Side by side
Compared on what you take on, rather than on which model scores higher this month.
| Dimension | Managed API | Self-hosted |
|---|---|---|
| Time to first working call | Minutes | Weeks, including hardware decisions |
| Who handles a traffic spike | The vendor | You, in advance, by buying capacity |
| Where the data goes | To a third party under contract | Nowhere you do not control |
| Model changes | The vendor may retire or update a version | Nothing changes until you change it |
| Cost shape | Per token, scaling with use | Mostly fixed, whether busy or idle |
| Skills needed | Application engineers | Application plus infrastructure engineers |
| Main failure | A vendor change you did not choose | Idle hardware, or a queue at peak |
| Sensible trigger | Almost every starting point | A rule that forbids sending the data out |
- CapacityBought before you know demand
- ServingSomething must batch the requests
- UpgradesNew weights mean new testing
- On callThe pager is now yours
- EvalsThe only proof quality held
None of these disappear at small scale. They just get shared with whoever has spare time.
Managed API
Where it wins
- You are building features on day one instead of choosing hardware.
- Capacity is somebody else's problem, including the week your product gets written about.
- Compliance paperwork already exists, which shortens a security review from months to weeks.
- Switching model size is a line of configuration rather than a procurement cycle.
Where it hurts
- Your data goes to a third party, and no contract makes that fact disappear.
- The vendor can change or retire a model version, and your prompts were tuned against the old one.
- Costs rise with success, which is fine until one high-volume feature dominates the bill.
- Zero data retention is an approved arrangement at the major vendors rather than a default, so plan for the application.
Self-hosted
Where it wins
- Data never leaves infrastructure you control, which some contracts and regulators simply require.
- The model stays exactly as it is until you decide otherwise, so behaviour does not drift under you.
- Steady heavy volume can suit fixed capacity better than per-token billing.
- Open-weight options are real: Mistral publishes several models under Apache 2.0, and Llama models come with a licence that permits commercial use.
Where it hurts
- You are now running an inference service, with the sizing, patching and on-call duty that implies.
- Idle hardware costs the same as busy hardware, so quiet periods are pure loss.
- Licences are not uniform. Llama requires a separate licence from Meta above 700 million monthly active users, and some Mistral models are not Apache 2.0 at all.
- You inherit safety work that a managed vendor was doing for you, including filtering and abuse handling.
How to choose
- Choose a managed API for anything you are still proving. Buying hardware to test an idea is an expensive way to learn the idea was wrong.
- Choose self-hosting if a contract, a regulator or a client's security team forbids the data leaving your network.
- Choose self-hosting if volume is high, steady and predictable, since that is the shape fixed capacity suits.
- Choose a managed API if your traffic is spiky. Paying for peak capacity all month to survive one hour is a bad trade.
- Choose both if one narrow workload is sensitive and the rest is not. Splitting by data classification is common and works.
- Choose neither yet if nobody can say what accuracy the system needs. Hosting decisions made before that answer get remade.
AI systems we keep running after launch
Questions teams ask before buying hardware
01Is self-hosting actually cheaper at scale?
It can be, once volume is high and steady enough to keep expensive hardware busy. The comparison people get wrong leaves out the engineers: sizing, serving, upgrades and on-call cover are ongoing work, not a one-off setup. Compare total running cost including those people, and be honest about how much of the day your hardware would sit idle.
02Can we start managed and move in-house later?
Yes, and that order is usually the right one. Keep model calls behind a single internal interface from the start, so the move is a swap rather than a rewrite. Build a scored evaluation set early too, because without one you cannot prove the in-house model performs as well as what you replaced.
03Does self-hosting solve our compliance problem?
It solves one part and creates another. The data stays inside your network, which is often the specific thing a rule requires. You then own everything the vendor was doing: access control, logging, retention, safety filtering and the evidence an auditor asks for. Self-hosting moves the obligation rather than removing it.
04How much hardware do we need?
Model size and peak concurrency decide it, so nobody can size your cluster responsibly from a page. The number that catches teams out is how many requests must run at the same moment, not average traffic. Measure your busiest realistic hour before buying anything. Peak decides the bill, and average decides the disappointment.
05Are open models good enough now?
For many narrow tasks, yes: classification, extraction, summarising within a known format. Broad reasoning across messy input is where the leading managed models generally remain ahead. Test on your own scored examples rather than on a public leaderboard, because leaderboard tasks are rarely shaped like yours.
