What is inference?
A demo that answers instantly for one tester can slow to a crawl once a thousand people are asking it things at the same time.
Inference
model serving
Inference is the step where a trained model processes a new input and produces an output, such as a chat reply or a classification. It is what runs every time a user sends a request. Training happens once, ahead of time; inference happens on every call after that.
One is the expensive, occasional event: feeding a model examples until its weights settle. The other is the cheap-looking, constant one: loading those settled weights and running a single request through them. A model gets trained a handful of times across its life and does inference millions of times.
That asymmetry is the whole point of the term. Nobody budgets monthly for training, because it does not happen monthly. Every serious AI budget is priced per request or per token, and it runs for as long as the feature stays live.
The cost you did not see in the demo
A prototype makes one call at a time, from one tester, on a quiet server. Nothing about that setup tells you what the feature costs or how fast it stays once fifty people use it together.
Two numbers decide whether inference survives that jump. Cost per request, because a call that seemed free at one-a-minute adds up fast at one-a-second. And latency under load, because a model that answers in two seconds alone can take twenty once requests queue behind each other on shared hardware.
Both are set by choices made before launch. Which model size the task actually needs. Whether responses get cached. And whether a smaller model handles the easy requests, leaving a larger one for only the hard ones. None of that shows up in a demo. All of it shows up in the first month of real traffic.
- 01Model size sets the floor on cost and latency before anything else is tuned.
- 02A prompt that grows with conversation history makes every later request in that thread more expensive than the first.
- 03Caching a repeated question avoids paying for inference twice on the same answer.
- RequestA prompt or an input arrives.
- LoadTrained weights, already in memory.
- ComputeForward pass, no learning happens.
- OutputReply, tokens, and their cost.
- RepeatEvery request pays this again.
Nothing here changes the model. Weights that get updated by feedback are a separate, much rarer process.
Related questions
01What is the difference between inference and training?
Training adjusts a model's weights using labelled examples, and happens rarely. Inference uses those fixed weights to answer a new request, and happens on every single call. One is a project. The other is an ongoing running cost.
02What does inference cost usually depend on?
Mainly the number of tokens processed and generated, and the model size doing the work. Longer prompts, longer replies and bigger models all raise the cost per request. Where the request runs, and whether it shares hardware with other traffic, affects it too.
03Why does inference get slower under load?
A single GPU can only run so many requests at once before they queue. The demo you tested ran alone. Production traffic arrives in bursts, and a request sitting in that queue takes longer to answer even though the model itself has not changed.
04Can inference cost change after launch without any code changes?
Yes. Model providers change prices, conversation history grows the prompt on every turn of a chat feature, and usage volume shifts with the business. An inference cost measured at launch is a snapshot, not a fixed number.
