Context Engineering Beats Prompt Engineering
Million-token context windows didn't kill retrieval. They moved the problem: from 'what can we fit' to 'what should be in there', which is a much harder question.
When context windows got long — genuinely long, hundreds of thousands of tokens and up — a lot of people concluded retrieval was dead. Just put everything in the prompt. No chunking, no embeddings, no vector database, no reranking.
I tried this. On a healthcare project with a real corpus and real latency requirements, I tried it seriously, because if it worked it would have deleted a lot of infrastructure.
It didn't work, and the reasons are more interesting than "the window wasn't big enough."
What long context actually changed
It did change things. Just not the thing people said.
Chunking got much less fussy. The elaborate chunking strategies of 2023 — semantic splitters, overlapping windows tuned to the token — mostly stopped mattering. You can retrieve whole documents now. If a relevant document is 8,000 tokens, send 8,000 tokens. Sentence-level precision is no longer worth the engineering.
Multi-document reasoning got real. Comparing across a dozen sources in one pass genuinely works now, and it didn't before. That unlocked a class of application.
The top-k tuning ritual died. Fetching 20 candidates instead of agonizing over 3 is now cheap enough to be the default.
What didn't change: you still have to decide what goes in. And that decision got harder, because the constraint stopped being mechanical and became a judgment call.
Why "just send everything" fails
Four reasons, in increasing order of how much they surprised me.
1. Cost and latency. The obvious one. Long context is priced per token, so a 400,000-token prompt on every turn is a business decision, not a technical convenience. It's also slow — time-to-first-token scales with input, and in any interactive system, especially voice, that's fatal. Prompt caching helps enormously when the bulk is stable. It helps not at all when the bulk is retrieved per-query, which is exactly the case here.
2. Attention isn't uniform. Models attend unevenly across long inputs. Bury the decisive fact in the middle of 300,000 tokens of adjacent material and recall degrades — not to zero, but enough to matter. This has improved steadily with each generation and it has never gone away. Position is still a variable you control, so control it: most-relevant material closest to the question.
3. Distractors actively hurt. This is the finding that changed how I build. Adding related but wrong content measurably degrades output — worse than adding nothing.
On the healthcare system, a patient query would pull the correct current record plus three superseded versions of overlapping information. The model would sometimes reason over a stale medication list. It wasn't a retrieval failure; the right document was right there. It was a precision failure. More context made the answer worse.
That's the crux. Retrieval isn't a workaround for small windows. Retrieval is how you exclude wrong information, and you need it regardless of how much fits.
4. You lose your debugging story. When something goes wrong with 15 retrieved documents, you can inspect them. With everything in context, "why did it say that?" has no tractable answer.
What I build now
The stack looks like RAG, but the emphasis has shifted from recall to precision and ordering.
Retrieve generously, filter hard. Pull 30–50 candidates. Then rerank and cut to what earns its place. The reranker matters more than the embedding model now — cheap recall, expensive precision.
Deduplicate and supersede aggressively. Near-duplicates and outdated versions are the main source of distractor damage. This is boring data-hygiene work and it delivered more quality improvement on that project than any prompt change:
function selectContext(candidates: Doc[], query: Query): Doc[] {
const fresh = dropSuperseded(candidates) // one version per entity, newest wins
const unique = dedupeByNearDuplicate(fresh, 0.92)
const ranked = rerank(unique, query)
return ranked.filter(d => d.score > THRESHOLD).slice(0, MAX_DOCS)
}
dropSuperseded was maybe 40 lines and fixed the stale-medication class of bug outright.
Order by relevance, not by however the database returned them. Highest-scoring material adjacent to the question. Free to implement, consistently measurable.
Structure beats prose. Rendering retrieved records as compact structured data rather than concatenated prose cut token count substantially and improved accuracy. Models are good at reading tables. Twelve documents of narrative text describing the same entity is a worse input than one table.
Cite and constrain. Every retrieved item carries an ID; the model must reference IDs in its answer. Unattributable claims become detectable, which turns "did it hallucinate?" from a vibe into a check you can run in CI.
Context as a budget you allocate
The reframe that made this click for me: context isn't storage you fill. It's a budget you allocate, and every token you spend has an opportunity cost — in money, in latency, and in attention diluted away from the tokens that matter.
So allocate deliberately. For an agent turn, roughly:
- Instructions — as small as they can be. Audit these; they accumulate contradictions.
- Tool definitions — tiered by task, not all of them, all the time.
- Session state — a distilled representation of what's established, not raw history.
- Retrieved knowledge — precision-filtered, ordered, structured, attributed.
- The current turn — last, so everything above it stays cacheable.
Note that four of those five are engineering problems, not writing problems. The prompt — the actual English instructions — is one line item, and usually not the one holding you back.
That's what I mean by context engineering beating prompt engineering. Not that wording is irrelevant. That once you're past the obvious mistakes in wording, essentially all remaining headroom is in what you put in front of the model and how you arrange it — which is a retrieval, state management, and data hygiene problem.
The version of "long context killed RAG" that's true
There is a real version of the claim, and I don't want to dismiss it.
If your corpus is genuinely small — a few hundred thousand tokens total, stable, no per-user access control — then yes: skip the vector database, send the whole thing, cache the prefix, and go build something else. That configuration is more common than RAG vendors would like you to think, and I've recommended it more than once.
But "my corpus fits" is a narrow condition. It fails as soon as you have per-tenant data, documents that change, access control, or anything at genuine scale. Which is most production systems.
The infrastructure didn't disappear. It moved up a layer, from "make it fit" to "make it right." That's a better problem to have, and a harder one.