Budget context as evidence

Treat context as a budgeted working set

A large context window is capacity, not proof that every included fact will influence the answer correctly. The original Lost in the Middle research showed positional sensitivity on multi-document question answering and key-value retrieval: relevant information could be used less reliably when placed inside a long prompt. That experiment is not a direct measurement of every current coding model, but it supplies a durable warning against indiscriminate packing. For repository work, reserve space for the task contract, governing instructions, current code, test evidence, and the agent's own intermediate outputs. Everything else competes for attention and cost. Set a target working set for each step instead of attaching the whole repository once and hoping the model will discover the hierarchy.

Retrieve in layers with provenance

Start from deterministic anchors: paths named in the issue, failing test identifiers, imports, symbols, stack frames, and ownership files. Expand to callers, configuration, and documentation only when the first layer leaves a stated question unanswered. Semantic search can help when terminology differs; GitHub documents repository indexing and semantic code search as context mechanisms. Search results remain candidates, not authority. Every retrieved chunk should retain its path, revision, and reason for inclusion. Mark generated files, vendored code, comments, tickets, and external pages as distinct trust classes. A concise context manifest lets a reviewer see what the agent did not read as well as what it did, which is essential when diagnosing a confident but incomplete patch.

Worked example: a hypothetical authorization bug

Suppose a hypothetical issue says that suspended accounts can still download invoices. A poor strategy retrieves every file mentioning account or invoice. A layered strategy begins with the failing route, its authorization middleware, the account-state type, and existing access tests. The agent records one open question: whether suspension blocks all exports or only new purchases. It then retrieves the policy document and two sibling endpoints that use the same state. Customer tickets are excluded because they contain personal data and are not needed to resolve the rule. The final context manifest lists eight files at one commit and labels the policy as governing evidence. If the policy remains ambiguous, the correct next action is clarification, not another broad retrieval pass.

Test retrieval, not just final answers

Create small probes with known evidence locations: a renamed symbol, a policy that conflicts with an old comment, a relevant fact deep in a long file, and a tempting unrelated match. Measure whether retrieval finds the authoritative item, preserves its revision, and excludes a protected directory. Then test whether the agent cites that evidence in its plan and detects conflicts. Tune lexical and semantic retrieval against misses and noise rather than a generic top-k setting. Include context freshness in the test; an index that trails the working tree can produce internally coherent but obsolete reasoning. Finally, define an escalation threshold: when evidence conflicts, exceeds the budget, or crosses a sensitive boundary, stop and narrow the question with a human.

Lost in the Middle paper ↗

GitHub repository indexing ↗

Learn from failed retrieval

Retain failed queries and the evidence eventually found. They expose vocabulary gaps, stale indexes, and ranking blind spots that a successful final answer can conceal. Review recurring misses by repository area and task type. Fix naming, documentation, or index coverage before compensating with ever larger context bundles.