Context is an engineering budget
Windows now hold hundreds of thousands of tokens, sometimes a million. That did not remove context engineering from agent work. It moved the decision from 'what fits' to 'what earns a place', and most teams have not noticed the job changed.
The short version
5 things that decide this
- 01A larger context window is capacity, not a plan. Filling it with everything available is still a design choice, and usually the wrong one.
- 02Models degrade at retrieving information buried in the middle of a long context, even when the answer is technically present in the window.
- 03Three levers do the real work: what goes in directly, what gets summarized before it goes in, and what stays out and gets fetched only on demand.
- 04Cost and latency scale with tokens sent, not tokens available. A million-token window used carelessly is a slow, expensive way to get a worse answer.
- 05Long context and retrieval are not rivals. Most production agents need both, aimed at different parts of the problem.
A bigger window moved the budget, not the need for one
Early agent builds hit a hard wall. A window of 4,000 or 8,000 tokens rarely held a full conversation, a document, and a tool result at once. Teams built retrieval and summarization out of necessity. Today's frontier models accept far more, and that changed what fails first.
It did not change how a model reads what you send it. A model paying attention across 200,000 tokens is not paying attention to each of them the way it would to 2,000. The instruction buried on page 40 of a dumped document set competes with everything around it, and it does not reliably win.
Answers go missing in the middle of a long context
Researchers have a name for this: lost-in-the-middle. A fact placed at the very start or very end of a long context gets retrieved reliably. The same fact placed in the middle gets missed at a meaningfully higher rate, even though nothing about the token itself changed.
For a support agent, that failure is invisible until a customer notices the answer is wrong. The refund policy was in the context. It just did not get weighed the way a person skimming the same document would weigh it. Passing a test with the right document present is not the same as passing it reliably in production.
Three lanes for information: direct, summarized, fetched
Every piece of information an agent might need falls into one of three categories, and the category should be a decision, not a default.
- 01Direct: the system prompt, the current task, and the two or three facts the model needs on every turn. Small and stable, worth the permanent token cost.
- 02Summarized: a long conversation history or a large document, compressed down to what the next step needs. This forces someone to decide what the summary must keep.
- 03Fetched on demand: pulled in by a tool call the moment the task needs it, then dropped.
- DirectSystem prompt, current task. Small and permanent.
- SummarizedHistory and prior results, compressed to what the next step needs.
- Fetched on demandRecords and documents, pulled only when the task needs them.
The same fact can sit in any of the three lanes. The mistake is putting everything in the first one because the window can technically hold it.
One document, deeply understood, is what long context is for
A long window earns its keep when a task reasons across one large, connected artifact: a full contract, a codebase module, a legal filing. Splitting that into retrieved chunks can break the links between its parts. A bigger window removes that trade-off cleanly.
The trade-off returns the moment the task needs one fact inside a pile of unrelated documents, not a connected read. That is a retrieval problem. Dumping the whole pile into the window is slower and costs more per call. Fetching the two documents that matter answers better, too.
Questions this raises
01Does a bigger context window remove the need for RAG?
No. A bigger window changes what fits, not how reliably a model finds the right fact inside what fits. Retrieval still wins when the task means locating a fact among many unrelated documents. It narrows what the model has to read first.
02Why does an agent miss information that is technically in its context?
Models retrieve information near the start or end of a long context more reliably than information in the middle. Researchers call this lost-in-the-middle. The token is present. Attention to it is not uniform across the window.
03What is the simplest way to start managing context on purpose?
Split what the agent might need into three buckets. What belongs in every call, what gets summarized down to the current step's needs, and what only gets fetched on demand. Most teams find most of their context belongs in that third bucket.
