Best LLM inference providers for open-weight models
Once you pick an open-weight model, something still has to serve it fast enough to answer before the user gets bored. These six are the buying layer under that decision.
The short answer
Groq wins on raw speed for models it hosts. Together AI and Fireworks AI win on breadth, fine-tune serving and staying close behind each new open-weight release. DeepInfra and Replicate sit lower on price, with less operational polish around it.
No provider wins all three axes. Groq's custom chips push token throughput far past GPU-based competitors, but its model catalogue is narrower and new releases land later. Together and Fireworks trade a little speed for a wider catalogue and managed fine-tuning.
We pick per client build. A voice agent lives or dies on latency. A batch summarisation job barely notices it.
How this ranking was made
Verified
Throughput and latency figures come from Artificial Analysis. It runs the same prompts against every provider's public endpoint and publishes tokens-per-second and time-to-first-token by model. We did not run our own benchmark suite for this page.
Pricing was read from each provider's own pricing page on 14 August 2026 and is linked in each entry. Model freshness was judged by how fast the provider had a major open-weight release live, using each one's changelog or announcement history.
General-purpose cloud GPU rental with no managed inference layer was excluded. Serving your own container on raw compute is a different decision, covered in our self-hosted comparison linked below.
- Tokens per second
- Output throughput on comparable models, per Artificial Analysis.
- Time to first token
- How long a user waits before anything appears, which matters most for chat and voice.
- Price per million tokens
- Published rate for a mid-size open-weight model, input and output blended.
- Model freshness
- How quickly a new open-weight release becomes available to call.
The six compared
Throughput from Artificial Analysis, pricing from each provider's published rate card, both checked on 14 August 2026.
| Provider | Shape | Strongest at | Pick it when |
|---|---|---|---|
| Groq | Hosted, custom LPU chips | Raw output speed | Latency is the feature, not a background concern |
| Together AI | Hosted, GPU-based | Catalogue breadth plus managed fine-tuning | You want one account for serving and fine-tuning |
| Fireworks AI | Hosted, GPU-based | Fast releases, custom model deployment | You are shipping fine-tuned or custom models to production |
| DeepInfra | Hosted, GPU-based | Low price per token | Cost per token matters more than shaving milliseconds |
| Replicate | Hosted, per-second billing | Running LLM, image and audio models from one account | Your workload mixes LLMs with image or audio models |
| Amazon Bedrock | Hosted, inside AWS | Sitting next to infrastructure you already run | Procurement, IAM and billing already run through AWS |
The ranking
Ordered by how many production apps each one suits as a default starting point.
- 01
Groq ↗
The fastest tokens per second on the models it hosts
Groq runs models on its own LPU chips rather than GPUs. Artificial Analysis benchmarks have repeatedly shown it producing several times more tokens per second than GPU-based providers on the same model. For a voice agent, or a chat interface where the user watches text arrive, that gap is the whole product.
The trade-off is catalogue size. Groq hosts a shorter, curated list of models. A brand-new release typically lands elsewhere first, and fine-tuned model serving is not its focus.
We reach for Groq where a slow response breaks the interaction: a phone agent, a live chat widget. Batch or offline work routes elsewhere.
Best for
- Voice agents and chat interfaces where response time is the product
- Apps already built on a model Groq hosts
Not for
- Apps that need a model on Groq's catalogue the day it releases
- Teams that need managed fine-tuning from the same provider
- Chip
- Custom LPU, not GPU
- Strongest metric
- Tokens per second, per Artificial Analysis
Wide catalogue plus fine-tune serving in one account
Together AI lists a broad catalogue: Llama, Qwen, DeepSeek and Mistral families, plus fine-tuning and dedicated endpoints under one account. That combination matters once a team moves past a stock model and starts adapting one to its own data.
Throughput on its GPU-based serving sits behind Groq's LPU numbers in Artificial Analysis benchmarks. New open-weight releases tend to appear here within days, sometimes hours, of launch.
It is a reasonable default when a client wants one provider relationship instead of splitting serving and fine-tuning across two vendors.
Best for
- Teams planning to fine-tune an open-weight model
- Apps that want a new release available fast, without switching providers
Not for
- Interfaces where Groq-level latency is the deciding factor
- Teams that only ever call one stock model and never fine-tune
- Deployment
- Hosted, GPU-based
- Fine-tuning
- Managed, same account as serving
Fast to new releases, built for custom model deployment
Fireworks is built for teams running their own fine-tuned or compressed models, beyond calling a stock one. It supports custom model upload alongside a standard catalogue. It has a track record of landing major open-weight releases within a day or two.
Artificial Analysis benchmarks place its throughput in a similar band to Together's, behind Groq's custom chips.
It earns its place once a model stops being the stock version: compressed, merged or fine-tuned. That case needs a serving layer built for it, not bolted onto one.
Best for
- Teams deploying a fine-tuned or quantised checkpoint, not a stock model
- Apps that need a new open-weight release live within a day or two
Not for
- Simple single-model apps with no plan to customise the weights
- Interfaces where Groq-level latency is the deciding factor
- Deployment
- Hosted, GPU-based, custom model upload supported
- Release speed
- Typically within a day or two of a major open-weight launch
The lowest price per token among providers we compared
DeepInfra's published rate card runs consistently below Together and Fireworks on comparable open-weight models. A workload calling a model millions of times a month, with no latency pressure, feels that gap on the bill.
The trade is polish. Dashboards, support and fine-tuning tools are thinner than Together's or Fireworks's, based on our own account setup and their public docs.
We reach for it on background jobs: batch classification, summarisation, anything nobody is watching in real time. That's where token count moves the budget.
Best for
- High-volume batch jobs where price per token dominates the decision
- Workloads with no strict latency requirement
Not for
- Interfaces where a slow or inconsistent response is visible to a user
- Teams that want managed fine-tuning from the same provider
- Deployment
- Hosted, GPU-based
- Strongest metric
- Price per million tokens
Per-second billing across LLMs, image and audio models alike
Replicate's catalogue is not limited to text models. Its documentation lists thousands of public models spanning image generation, audio and video alongside LLMs. Most are billed by compute time, not a fixed per-token rate.
That range suits a product mixing model types, generating an image and then describing it, say, without a second vendor relationship. It is not the fastest or cheapest option for pure LLM throughput here.
Skip it if the workload is text-only. A provider built just for LLM serving will beat it on speed and price for that narrower job.
Best for
- Products calling LLM, image and audio models from one account
- Prototyping against a wide range of public models quickly
Not for
- Pure LLM workloads optimising purely for tokens per second or price
- Teams that need predictable per-token pricing for budgeting
- Billing
- Per-second compute time on most models
- Catalogue
- LLMs plus image, audio and video models
Open-weight serving inside infrastructure you already run
Bedrock serves a curated set of open-weight models, Llama among them, alongside proprietary ones, inside AWS's own access controls and billing. For a team whose procurement already runs through AWS, that removes a second vendor to onboard.
It lags the other providers on freshness, though. A new open-weight release typically reaches Bedrock after it is already live on Together, Fireworks or Groq.
It ranks last here on freshness and speed, not on quality. Where a client's compliance team only clears vendors already on their AWS bill, that rule outweighs the ranking.
Best for
- Teams whose security and procurement review is scoped to AWS already
- Apps that need serving, IAM and billing in one console
Not for
- Apps that need the newest open-weight release on day one
- Teams optimising purely for tokens per second
- Deployment
- Hosted, inside AWS
- Fits best
- Alongside an existing AWS deployment
- Your promptSent to each provider's public endpoint.
- Time to first tokenHow long before anything appears.
- Tokens per secondHow fast the rest streams in.
- Price per million tokensWhat the same request costs across providers.
- Your productFeels the first two numbers. Your bill feels the third.
Artificial Analysis publishes all three per model, per provider, refreshed as new benchmarks run.
“They have a problem-solving mindset, analytical skills, and deep technical knowledge.”
David Manley · CEO, Go Real Travel
When none of these is the right layer
If your app calls a closed model like Claude or GPT, this list is not for you. Those are served directly by their maker, and no third-party provider hosts them.
And if request volume runs high and steady, renting raw GPUs can beat any of these on cost per token. The price is owning the uptime yourself. That break-even point sits higher than most teams expect.
- 01Calling a proprietary model, not an open-weight one? None of these apply.
- 02Volume high and steady enough to justify your own GPU fleet? Compare against self-hosting first.
- 03Picking the model itself is a separate decision from picking who serves it.
Not sure which axis matters for your build?
We size the serving layer to the product's actual latency and volume needs, not to whichever provider benchmarks best in general. Scoping calls cost nothing.
Questions teams ask
01Which LLM inference provider is fastest?
Groq, on the models it hosts. Its custom LPU chips have repeatedly shown several times the tokens-per-second of GPU-based providers in Artificial Analysis benchmarks. The catalogue is narrower than Together's or Fireworks's, though.
02Which provider is cheapest for Llama hosting?
DeepInfra's published rate card runs below Together and Fireworks on comparable open-weight models, based on pricing checked 14 August 2026. Confirm current rates on each provider's pricing page before committing, since these change often.
03Do I need Together and Fireworks, or just one?
Most teams need one. Both cover a wide open-weight catalogue and managed fine-tuning. Pick whichever had your specific model live first, or whichever fine-tuning workflow fits your data pipeline better.
04Why is Amazon Bedrock ranked last?
Not on quality. Bedrock tends to lag the specialised providers on how fast a new release lands, and its throughput sits behind Groq's. It still wins when a client's compliance review only clears vendors already on their AWS account.
05Does a fast inference provider replace choosing the right model?
No. This page ranks who serves a model quickly and cheaply. Which model to serve is a separate decision, covered in our open-weight model ranking linked above.
Related reading
- Best open-weight LLMs →The model decision this page's providers all sit underneath.
- Open-weight vs API LLMs →Whether to leave the API model at all, before picking who serves the replacement.
- Self-hosted vs managed LLM infrastructure →The break-even point where running your own GPUs beats any provider on this page.
