Best LLM gateways for production apps in 2026
A gateway sits between your app and every model provider. Before picking one, check whether your app is calling more than one provider at all.
The short answer
LiteLLM is the strongest choice when you want the gateway inside your own infrastructure. OpenRouter is the strongest choice when you want provider breadth with nothing to run. Neither is the right answer for an app calling a single provider with no failover need.
A gateway earns its place once you need to fail over between providers, enforce spend limits across teams, or swap models without redeploying. Below that bar, it is a network hop and a bill you don't need yet.
We make this call per client build, based on how many providers the app actually talks to and who has to answer for an outage.
How this ranking was made
Verified
We ranked on what happens when a provider fails or slows down. That is the moment a gateway earns its keep, or adds a second point of failure. Ease of the first integration mattered less than what breaks at month six.
Capabilities were read from each project's own documentation on 14 August 2026 and are linked in each entry. We run direct provider integrations in client production systems, and add a gateway only where the failover or governance need is real.
General-purpose API gateways with no LLM-specific routing were excluded. Token-aware retries and per-model rate limits are the entire point of this category.
- Failover behaviour
- What happens automatically when a provider errors, rate-limits or times out.
- Added latency
- The extra hop a gateway puts between your app and the model.
- Observability
- Whether you can see cost and errors per model without building it yourself.
- Data control
- Where request and response bodies travel, and who can see them.
The five compared
Read from vendor and project documentation on 14 August 2026.
| Option | Shape | Failover | Pick it when |
|---|---|---|---|
| No gateway, direct SDK | Your app calls the provider's own SDK | You write it, or there is none | One provider, no routing decision to make |
| LiteLLM | Self-hosted proxy | Configurable, ordered fallback chains | You want the routing layer inside your own infrastructure |
| OpenRouter | Hosted gateway | Automatic provider fallback per model | You want breadth across models with nothing to run |
| Portkey | Hosted gateway, open-source core | Configurable fallback plus guardrails | You need budget limits and audit logging across teams |
| Kong AI Gateway | Self-hosted, plugin-based | Configurable via plugin chain | LLM traffic sits beside APIs you already run through Kong |
The ranking
Ordered by how many production apps each one suits as a starting point.
- 01
No gateway, direct provider SDK
The honest first entry for a single-provider app
This ranks first because it is what most apps should still be running. Call one provider with no outage story yet, and a gateway just adds a network hop, a new bill and a new failure point. That is a high price for a failover path you have never tested.
Anthropic's and OpenAI's own SDKs already handle retries on transient errors. A gateway's real value shows up when you need to route between two or more providers, not before.
We build client systems this way by default. A gateway gets added only when a second provider enters the picture, or spend controls across teams become a real requirement.
Best for
- Apps built on a single provider with no near-term plan to switch
- Teams who want the fewest moving parts in production
Not for
- Apps that need to fail over to a second provider automatically
- Multiple teams sharing one budget across several models
- Added latency
- None
- Extra infrastructure
- None
Self-hosted proxy, one OpenAI-compatible endpoint
Choose this when you want the gateway running inside your own infrastructure, not a third party's. LiteLLM's documentation describes ordered fallback chains. A request tries the first deployment, and on a connection error, a 404 or a 429 it moves to the next one you configured. Your application code never changes.
It also documents round-robin, least-busy, usage-based and latency-based load balancing across keys or providers. Virtual keys and per-team budget limits track to your own Postgres database, not a vendor's.
The cost is operational. You run the proxy, patch it, and own the incident when it goes down. That trade only makes sense once you have someone who can carry that pager.
Best for
- Teams that want routing logic and cost data inside their own systems
- Multi-provider apps where request bodies cannot leave your infrastructure
Not for
- Teams with nobody to operate a proxy in production
- Apps that just need model breadth with no governance need
- Deployment
- Self-hosted
- Fallback
- Ordered chains, configurable
Hosted gateway across 400-plus models
Pick this when breadth matters more than owning the infrastructure. OpenRouter's documentation lists routing to models from 60-plus providers behind one OpenAI-compatible API. Provider-level failover is on by default, and model-level fallback triggers when you pass a priority list.
Its pricing passes through provider rates with no markup, plus a fee on credit purchases, and it documents that a failed request is not billed. Fallback firing also means you pay the rate of whichever model actually served the response.
The trade-off is that your traffic passes through a hosted third party. Experimentation and apps that switch models often can accept that exchange. Data under strict rules needs a closer read of their data policy filters first.
Best for
- Apps that need access to many models with no infrastructure to run
- Teams still deciding which model fits, who want to switch without redeploying
Not for
- Traffic that cannot pass through a third-party host at all
- Teams that already know their one or two providers and want fewer hops
- Deployment
- Hosted
- Provider count
- 60-plus, per OpenRouter's own documentation
Hosted gateway built for team-level governance
Reach for this when the problem is not routing but oversight: several teams, several budgets, one person who has to explain the bill. Portkey documents role-based access control, per-key budget limits and audit logging on its enterprise tier. Routing and fallback come standard underneath that.
Its gateway is open-source at the core. Both hosted and self-hosted deployment are documented, so the governance layer does not force one deployment model.
Skip it if your actual problem is model breadth or a single team's routing. The governance features are the reason to pick this one, and they cost nothing to a team that does not need them.
Best for
- Multiple teams sharing model access under one budget
- Organisations that need an audit trail on who called what
Not for
- A single small team with one shared key already
- Apps optimising purely for lowest added latency
- Deployment
- Hosted or self-hosted
- Governance
- RBAC, budgets, audit logging on enterprise tier
- 05
Kong AI Gateway
LLM routing as a plugin on infrastructure you already run
This is the pick when LLM calls are one traffic type among many and you already run Kong for the rest of your APIs. It adds LLM-specific plugins to routing and policy enforcement you have already invested in, rather than standing up a second gateway.
Fallback and rate limiting run through its plugin chain, the same mechanism used for every other route. That's one operational model instead of two.
Starting from zero for an LLM-only app, this is more infrastructure than the problem needs. It earns its place by extending something already in production, not by being simplest to stand up.
Best for
- Platform teams already running Kong for general API traffic
- Organisations that want one policy engine across all traffic types
Not for
- A greenfield app with no existing API gateway
- Small teams who want the fastest path to a first integration
- Deployment
- Self-hosted
- Fits best
- Alongside an existing Kong deployment
- Your appSends the request.
- GatewayPicks a provider, applies limits.
- Provider fails or limitsThe moment a gateway earns its keep.
- Fallback firesNext provider in the chain, same request.
- ResponseYour app never sees the retry.
Skip the middle three steps and you have the direct-SDK option: fewer parts, no fallback.
When a gateway is the wrong call
A gateway cannot fix a design problem. If one model going down already means your product is down, routing between providers buys you very little. Your prompts, your evals and your output format are still tuned to one model's behaviour.
Latency is the other cost that gets skipped over. Every hop through a hosted gateway adds network time on top of the model's own response time. On a chat interface where a user is watching the cursor, that adds up.
- 01Calling one provider with no plan to add a second? Skip the gateway.
- 02Fallback you have never tested is not a resilience plan, it's an assumption.
- 03If latency is the metric you are protecting, measure the added hop before you add it.
“They have a problem-solving mindset, analytical skills, and deep technical knowledge.”
David Manley · CEO, Go Real Travel
Deciding whether your app needs a gateway at all?
We size the routing layer to the failure modes an app actually has, not to what the category sells. Scoping calls cost nothing.
Questions teams ask
01Do I need an LLM gateway if I only use one provider?
Usually not. A gateway earns its place once you route between providers, enforce budgets across teams, or need a fallback path you have actually tested. A single-provider app with one team gets little from the extra hop.
02Is OpenRouter or LiteLLM the better starting point?
OpenRouter if you want hosted breadth across many models with nothing to run yourself. LiteLLM if you want the routing and cost data to live inside your own infrastructure. Teams with strict data rules tend to land on LiteLLM.
03How much latency does a gateway add?
A self-hosted proxy on infrastructure close to your app adds very little. A hosted gateway adds a network hop to a third party, on top of the model's own response time. That matters most on latency-sensitive interfaces like chat.
04Why did you leave out Kong until fifth?
Kong AI Gateway is a strong pick, but only for teams already running Kong for other API traffic. Starting from zero, it is more infrastructure than most LLM-only apps need, which is why it ranks below options built specifically for this problem.
05Does a gateway replace evaluation or monitoring?
No. A gateway routes and enforces limits. It does not tell you whether the model's answer is correct. Evaluation is a separate layer, and skipping it because a gateway is in place is a common mistake.
