OpenAI API
The first call takes ten minutes. Production takes the rest
Nothing about calling the API is hard. What costs you a sprint is what happens on the day traffic doubles and the API starts returning 429.
The verdict
The OpenAI API is a sound default for text generation, classification and extraction in a product, provided you design for its tiered rate limits from the first week rather than discovering them under load.
The models are strong and the SDKs are good. Neither is the reason projects struggle. Throughput planning is.
Treat rate limits as an architectural input, the same way you would treat a database connection pool. They decide whether your feature is synchronous or queued.
In short
5 things that decide this
- 01Limits apply across several dimensions at once, including requests per minute, requests per day, tokens per minute and tokens per day.
- 02Exceeding a limit returns a 429 error, and the documented remedy is retrying with exponential backoff.
- 03The official SDKs automatically retry eligible rate-limit errors and honour Retry-After, so hand-rolled retry logic often duplicates work already done.
- 04Response headers report remaining requests and tokens, which is how you build a throughput dashboard instead of guessing.
- 05Usage tiers raise your limits as your account matures, so the ceiling you test against is not the ceiling you launch with.
For someone who has not shipped on it
You send text and a model returns text. That is the whole interface, and it is why a first working version takes an afternoon.
The operational surface is the part to plan. Requests are metered on more than one axis. Failures are normal rather than exceptional. And the model behind a name can change unless you pin a version.
What we have built with it
Seven of our shipped products call the OpenAI API, and they use it for different jobs. Broollie runs GPT-4o and GPT-4o-mini across a multi-tenant meeting platform in more than 38 countries. Greenlight reads laws, certifications, studies and controversies to score how sustainable a company really is, covering more than 50 ESG topics with 10 to 15 independent sources each.
Little Tree Confections is the smallest and clearest. Meeting transcripts flow through an OpenAI refinement step inside n8n, then become ClickUp tasks and Notion docs. Golancer generates daily priorities for freelancers, and ZhoopZhoop uses it in voice agents answering an auto repair shop's phone.
The pattern across all seven: the model call is a small part of the system. Retrieval, validation and the queue around it are where the engineering time actually goes.
- CallThe easy ten minutes.
- LimitRequests and tokens, per minute.
- 429Back off, then retry.
- PinName the version you tested.
- LogPrompt, model, tokens, reply.
- ScoreEvals gate the next change.
Every station after the first is where the sprint goes, and skipping the last two is why AI features quietly get worse.
Where it stands
Good at
- Time to a working prototype is unmatched, which makes it the fastest way to prove a feature is worth building properly.
- The official SDKs handle retries for eligible rate-limit errors and respect Retry-After, so the common failure is handled for you.
- Rate-limit headers expose remaining requests and tokens, which turns capacity planning into something you can chart.
- A batch path exists for work that does not need an answer now, which is the right home for bulk processing.
Weak at
- Rate limits bind on several dimensions at once, so a system comfortably inside its request budget can still fail on tokens per minute.
- Your limits change as your account tier changes, which means load testing early gives you a number that will not match production.
- Latency is dominated by token generation, so a chatty response format costs you response time as well as money on every single call.
- Behaviour moves when the model behind a name moves, and without a pinned version and a scored test set you find out from users.
- It is an external dependency in your critical path, so your feature's availability is now partly somebody else's operational record.
Systems running model calls in production
Common questions
01How do we stop hitting rate limits?
Shape the traffic before you ask for more capacity. Queue anything a user is not actively waiting for, cap output length, and cache answers to questions that repeat. Watch the rate-limit headers for remaining requests and tokens so you see the ceiling approaching rather than meeting it.
02Should we write our own retry logic?
Usually not, because the official SDKs already retry eligible rate-limit errors and honour Retry-After. Adding your own layer on top produces double retries and a thundering herd during an incident. Read what your SDK does before writing anything.
03How do we keep behaviour stable over time?
Pin a specific model version and keep a scored test set. A version you chose does not change under you, which turns an upgrade into a decision rather than a surprise. The test set is what tells you whether the next version is safe to move to.
04What should we log on every call?
The prompt version, the model name, the tokens used and the reply, tied to the request that caused it. Cost questions, quality complaints and incident reviews all resolve from that record. Teams that log only the reply cannot answer any of the three.
