Best LLM gateways (2026)
A gateway sits between your app and every model provider. Before picking one, check whether your app is calling more than one provider at all.
The short answer
LiteLLM is the strongest choice when you want the gateway inside your own infrastructure, and OpenRouter when you want model breadth with nothing to run. A single-provider app with no failover need should skip both.
A gateway earns its place once you need to fail over between providers, enforce spend limits across teams, or swap models without redeploying. Below that bar, it is a network hop and a bill you don't need yet.
We make this call per client build, based on how many providers the app actually talks to and who has to answer for an outage.
Key takeaways
3 things that decide this
- 01Most single-provider apps should still call the provider's SDK directly; LiteLLM and OpenRouter lead once you route between providers.
- 02Four criteria set the order: failover behaviour, added latency, observability and data control.
- 03Test your fallback path on purpose before you rely on it, and decide where your request bodies are allowed to travel.
How we ranked these tools
Verified
We ranked on what happens when a provider fails or slows down. That is the moment a gateway earns its keep, or adds a second point of failure. Ease of the first integration mattered less than what breaks at month six.
For every option we checked its own documentation for fallback behaviour, routing and load balancing, budget and key controls, deployment model and data handling. We checked all of it on 11 November 2026, and each entry links to its source.
General-purpose API gateways with no LLM-specific routing were excluded. Token-aware retries and per-model limits are the entire point of this category.
Hashlogics publishes this page. We build custom software and are not one of the tools ranked here. No company paid for its place.
- Failover behaviour
- What happens automatically when a provider errors, rate-limits or times out.
- Added latency
- How much extra time the gateway's hop adds between your app and the model.
- Observability
- Whether you can see cost and errors per model without building it yourself.
- Data control
- Where request and response bodies travel, and who can see them.
The five compared
Read from vendor and project documentation on 11 November 2026.
| Option | Shape | Failover | Pick it when |
|---|---|---|---|
| No gateway, direct SDK | Your app calls the provider's own SDK | You write it, or there is none | One provider, no routing decision to make |
| LiteLLM | Self-hosted proxy | Configurable, ordered fallback chains | You want the routing layer inside your own infrastructure |
| OpenRouter | Hosted gateway | Automatic provider fallback per model | You want breadth across models with nothing to run |
| Portkey | Hosted gateway, open-source core | Fallbacks between providers and models | You need budget limits and access control across teams |
| Kong AI Gateway | Self-hosted or Kong-hosted | Load balancing with failover across providers | LLM traffic sits beside APIs you already run through Kong |
The ranking
Ordered by how many production apps each one suits as a starting point.
- 01
No gateway, direct provider SDK
The honest first entry for a single-provider app
This ranks first because it is what most apps should still be running. Call one provider with no outage story yet, and a gateway just adds a network hop, a new bill and a new failure point. That is a high price for a failover path you have never tested.
Anthropic's and OpenAI's own SDKs already handle retries on transient errors. A gateway's real value shows up when you need to route between two or more providers, not before.
We build client systems this way by default. A gateway gets added only when a second provider enters the picture, or spend controls across teams become a real requirement.
Best for
- Apps built on a single provider with no near-term plan to switch
- Teams who want the fewest moving parts in production
Not for
- Apps that need to fail over to a second provider automatically
- Multiple teams sharing one budget across several models
- Added latency
- None
- Extra infrastructure
- None
Self-hosted proxy, one OpenAI-compatible endpoint
Choose this when you want the gateway running inside your own infrastructure, not a third party's. LiteLLM's router tries the next priority tier when a deployment fails with a connection error, a 404 or a 429. Your application code never changes.
It also documents simple-shuffle, least-busy, usage-based and latency-based routing across keys or providers. It warns that usage-based routing isn't recommended for production. Virtual keys carry per-key and per-team budgets, tracked in your own Postgres database.
Your cost is operational. You run the proxy, patch it, and own the incident when it goes down. That trade only makes sense once you have someone who can carry that pager.
Best for
- Teams that want routing logic and cost data inside their own systems
- Multi-provider apps where request bodies cannot leave your infrastructure
Not for
- Teams with nobody to operate a proxy in production
- Apps that just need model breadth with no governance need
- Deployment
- Self-hosted
- Fallback
- Priority tiers, configurable
Hosted gateway across hundreds of models
Pick this when breadth matters more than owning the infrastructure. OpenRouter describes hundreds of models from many providers behind one API key. If a provider returns an error, it falls back to the next provider automatically.
Its FAQ says there's no markup on inference pricing, which passes through provider rates, and that a fee applies to credit purchases. Fallback firing also means you pay the rate of whichever provider actually served the response.
The trade-off is that your traffic passes through a hosted third party. Its privacy settings can exclude providers that log data, or keep you on zero-data-retention providers only. Read those before you send data under strict rules.
Best for
- Apps that need access to many models with no infrastructure to run
- Teams still deciding which model fits, who want to switch without redeploying
Not for
- Traffic that cannot pass through a third-party host at all
- Teams that already know their one or two providers and want fewer hops
- Deployment
- Hosted
- Models
- Hundreds, per OpenRouter
Hosted gateway built for team-level governance
Reach for this when the problem isn't routing but oversight: several teams, several budgets, one person who has to explain the bill. Portkey's gateway supports fallbacks between providers and models, and lets you set usage limits on cost or tokens. Its Enterprise plan lists role-based access control, SSO and granular budget and rate limits.
Portkey's gateway is open-source at the core. You can self-host it and connect it to Portkey, so governance doesn't force one deployment model on you.
Skip it if your actual problem is model breadth or a single team's routing. Governance is the reason to pick this one.
Best for
- Multiple teams sharing model access under one budget
- Organisations that need an audit trail on who called what
Not for
- A single small team with one shared key already
- Apps optimising purely for lowest added latency
- Deployment
- Hosted or self-hosted
- Governance
- RBAC, SSO and budgets on Enterprise
LLM routing as a plugin on infrastructure you already run
This is the pick when LLM calls are one traffic type among many and you already run Kong for the rest of your APIs. Kong AI Gateway routes requests across providers such as OpenAI, Anthropic, Azure AI, Amazon Bedrock and Gemini. Its load balancing fails over when a provider is slow or down.
Spend and token limits run through AI policies, scoped by team through consumer groups. In AI Gateway 2.x, AI Models and AI Model Providers replace the older plugin model. You can self-host it on Kong Gateway or use Kong's hosted gateways.
Starting from zero for an LLM-only app, this is more infrastructure than the problem needs. It earns its place by extending something already in production.
Best for
- Platform teams already running Kong for general API traffic
- Organisations that want one policy engine across all traffic types
Not for
- A greenfield app with no existing API gateway
- Small teams who want the fastest path to a first integration
- Deployment
- Self-hosted or Kong-hosted
- Fits best
- Alongside an existing Kong deployment
- Your appSends the request.
- GatewayPicks a provider, applies limits.
- Provider fails or limitsThe moment a gateway earns its keep.
- Fallback firesNext provider in the chain, same request.
- ResponseYour app never sees the retry.
Skip the middle three steps and you have the direct-SDK option: fewer parts, no fallback.
When a gateway is the wrong call
A gateway cannot fix a design problem. If one model going down already means your product is down, routing between providers buys you very little. Your prompts, your evals and your output format are still tuned to one model's behaviour.
Latency is the other cost that gets skipped over. Every hop through a hosted gateway adds network time on top of the model's own response time. On a chat interface where a user is watching the cursor, that adds up.
A gateway is plumbing. Deciding what routes through it, and what happens when a provider fails, is system design — the kind our AI development team handles.
- 01Calling one provider with no plan to add a second? Skip the gateway.
- 02Fallback you have never tested is not a resilience plan, it's an assumption.
- 03If latency is the metric you are protecting, measure the added hop before you add it.
“They have a problem-solving mindset, analytical skills, and deep technical knowledge.”
David Manley · CEO, Go Real Travel
Deciding whether your app needs a gateway at all?
We size the routing layer to the failure modes an app actually has, not to what the category sells. The scoping call is free. Every build then starts with a fixed-fee Blueprint, credited in full against the build, and if the plan isn't one you'd act on, you don't pay for it.
Questions teams ask
01What is the best LLM gateway?+
LiteLLM is the best LLM gateway if you want routing inside your own infrastructure, and OpenRouter if you want hosted breadth across models. If you call one provider with no failover need, the best gateway is none.
02How do I compare LLM gateways fairly?+
Break one provider on purpose in a test and watch what each gateway does. Then compare the added latency on your own traffic, what cost data you can see per model, and where request bodies travel.
03Do I need an LLM gateway if I only use one provider?+
Usually not. A gateway earns its place once you route between providers, enforce budgets across teams, or need a fallback path you have actually tested. A single-provider app with one team gets little from the extra hop.
04Is OpenRouter or LiteLLM the better starting point?+
OpenRouter if you want hosted breadth across many models with nothing to run yourself. LiteLLM if you want the routing and cost data to live inside your own infrastructure. Teams with strict data rules tend to land on LiteLLM.
05How much latency does a gateway add?+
A self-hosted proxy on infrastructure close to your app adds very little. A hosted gateway adds a network hop to a third party, on top of the model's own response time. That matters most on latency-sensitive interfaces like chat.
06Why did you leave out Kong until fifth?+
Kong AI Gateway is a strong pick, but only for teams already running Kong for other API traffic. Starting from zero, it is more infrastructure than most LLM-only apps need, which is why it ranks below options built specifically for this problem.
07Does a gateway replace evaluation or monitoring?+
No. A gateway routes and enforces limits. It does not tell you whether the model's answer is correct. Evaluation is a separate layer, and skipping it because a gateway is in place is a common mistake.
08How were these tools ranked?+
We scored four gateways and the direct-SDK option on failover, added latency, observability and data control, checked on 11 November 2026. Hashlogics publishes this page, builds custom software, and is not one of the tools ranked.
