Does indexing your codebase actually help?

The question, stated so it can fail

Given a bug report, does the model name a file that the real fix actually touched — and does having an index of the repository make that more likely than not having one?

Ground truth is not opinion. Each case is a public issue, the pull request that closed it, and the files that pull request changed. The primary metric is whether the arm wrote down a path the fix touched. It is checkable, and a model cannot argue its way to a pass.

How it was run

Four arms. The same model throughout, the same prompts throughout. They differ only in what context they receive.

BLIND is the one that matters. Every public repository here is in the model’s training data, so it can name a file from memory with no index at all. Hiding the repository name is the closest available proxy for a private codebase — and it is only a proxy, which is why a private repository was measured separately.

Leakage controls

Results — public repositories

Share of honest-subset cases where the arm named a file the fix touched. 117 cases across 8 repositories. Held-out cases were never tuned against; the last column is repositories introduced after all tuning had finished.

Paired on identical cases

Percentages across arms can move for reasons that have nothing to do with the index. Pairing each arm against the other on the same case removes that: every row below is the same tickets, answered twice.

On repositories introduced after all tuning, ON beats BLIND 12–0 and the gagged OFF arm scores exactly zero. The gap does not narrow as cases move further from anything tuned against, which is the opposite of what overfitting looks like.

Method note: two cases in this run were re-run individually — one lost its connection mid-request, one was served a fallback template instead of model output. Both were caught mechanically by the guard that fails any run containing template text, and both were replaced with real output rather than dropped. Excluding them would have biased the set toward short prompts, since a timeout is likelier on a long one.

Results — a private repository the model has never seen

The public result has one unavoidable weakness: those repositories were in training data. So the same measurement was run on a private repository — 409 files, Node/Express + SQLite — with cases built by two agents that could not communicate.

• A planter introduced one realistic single-file regression per case — a swapped INSERT argument, || where ?? was needed, an off-by-one on inclusive dates — recorded ground truth, and wrote no user-facing text at all.
• A reporter received only the mechanically sanitised runtime failure, with endpoints, UUIDs and status lines stripped. No repository access, zero tool calls. It wrote each ticket as the affected administrator would.

Neither could write toward the answer. That is a stronger guarantee than the public set, which relies on the issue predating the fix. The index is pinned to real main, so the mutated line is absent from it — ON cannot find the bug by spotting an anomalous edit, it has to localise from the reported symptom alone.

ON holds at ~50% on code no model has seen, while every baseline collapses to zero. They were not declining to answer. NEUTRAL and BLIND each cited paths on 9 of 12 tickets — one cited twelve paths on a single ticket — and were wrong every time, inventing plausible file layouts:

The last row is the giveaway: a Django layout guessed for an Express codebase. The concern going in was that user vocabulary — “leave balance”, “payslips” — maps to filenames by convention and hands the no-index arms a free shortcut. It did not happen. Naming the resource is worthless without knowing the repository is Express with a backend/src/controllers/ layout, and that is exactly what the index supplies.

Scope of this run, so the number is read against the right thing: 12 cases, one repository, 409 files, Node/Express + SQLite, one planted single-file regression per case, across twelve distinct files — payroll, leave, attendance, reports, settings, org structure, shifts, holidays, approvals and projects.

How much a single run can be trusted

The end-to-end metric is noisier than it looks, and this was measured rather than assumed — after a claim had already been made on the strength of one run. The same 34cases, the same code, the same index, in two runs differing only in the model’s own choices:

A 5-case swing. The explore loop picks its own search terms, so identical inputs do not produce identical retrieval. Any single-run difference below roughly 5 cases at n≈34 — or 3 at n≈81 — is indistinguishable from that variance.

Applied throughout: differences inside that band are reported as unchanged, never as small wins. The ON-versus-baseline gaps sit far outside it — 30–1 paired, p<0.001 — which is why the headline stands while smaller effects on this page do not.

A retrieval change that did not pay off

Symbol indexing was added late. Symbols existed for only about two dozen summarised files per repository, while the index actually being searched held every path — zero of directus’s 3,955 entries carried one. Wiring them in improved the deterministic ranker substantially, and that measurement involves no model and no randomness, so it reproduces exactly:

It did not convert end-to-end. Held-out ON moved 48.1% → 50.6% — two cases, inside the noise band above — and the private set was unchanged at exactly 6/12 before and after.

The reason is visible in the data. The candidate paths come from the explore loop’s own distilled queries, not from ranking the raw ticket text, so the agent was already compensating for weak ranking by grepping and re-querying. Improving the ranker improved a stage that was not the bottleneck. The remaining misses keep the same shape: 40 of 40are “retrieval never offered the file”, 0are “offered and ignored”.

Reported because it is a real result: an improvement that is genuine at one layer and invisible at the layer that matters.

Judges, and why the factual metric leads

Two judges from different vendors scored the same stored answers, blind — one from the same model family as the system under test, one from an unrelated family. Self-preference was the worry. The opposite occurred: the foreign judge was more favourable to the indexed arm.

Judge deltas are noisy, so they are published as ranges

Three identical re-runs of the same judge over the same stored answers do not return the same number. One comparison spanned 0.14 across passes — wider than most of the differences anyone would want to read into a single run.

Direction never flips — every judge, every pass, puts the indexed arm ahead. But a single delta quoted to two decimals would be over-precise, so none is quoted that way anywhere on this site.

Agreement between judges is below the noise floor on BLIND

Cohen’s κ on the winner label. The left column is a judge against itself, which sets the ceiling any cross-vendor agreement could reach.

On BLIND, cross-vendor agreement (κ≈0.23–0.32) sits clearly below the same-judge floor (κ≈0.55). There is a real family effect there, not just run-to-run variance. That is a reason to lead with the factual metric rather than to explain the discrepancy away.

Both judges penalise hedging

On the 40 cases where both arms are factually wrong — a bucket fixed by citation correctness in code, so it is the same 40 cases for both judges — both scored the no-index arm higher, by 0.42 on the mean. Their own rubric says a confidently wrong answer is worse than admitting uncertainty; both do the reverse. ON hedges when retrieval finds nothing, and is marked down for it.

The narrow version of that claim is the defensible one: both judges show a population-level tilt toward the no-context arm — significant for one (p=0.029), suggestive for the other (p=0.087). They do not independently confirm the same per-case phenomenon: per-case gap correlation is r=0.36 and they agree on direction in only 15/40 cases. The matching 0.42 is a coincidence of means.

Either way the direction holds, and it runs against ON. Judge scores understate the index relative to the factual metric, which is why the factual metric leads.

What it cost to run

Measured from token counts rather than estimated from case counts: ≈2.6 cents per case, where a case runs two arms. A production ticket runs the ON arm only, so the real per-ticket figure is lower. Input is 95% of spend. The gate run was 3,820,467 input and 183,740 output tokens over 1,414 calls across 117 cases.

Prompt caching currently returns 26.4%, and the reason is structural rather than mysterious: the constant prefix shared by every case is ~805 tokens and the provider only caches prefixes at ≥1,024. Nothing is shared between tickets; only later turns within one ticket hit, once the accumulated conversation crosses the threshold. It is a known, quantified gap worth about 58% of the bill.

The dollar figures are estimates from measured tokens at published rates, not billed amounts — the identity running the benchmark has no Cost Management access.

Reproducing this

The harness and the frozen dataset are committed in the backend repository under backend/benchmarks/, at a pinned SHA for the code under test. The pre-registered filters are frozen in config.mjs, and the runner works against a benchmark-only database — it never touches production data. Every repository is indexed at its 2026-05-01 freeze commit, and the run aborts if an indexed SHA does not match.

Honest record of what went wrong

Findings withdrawn or corrected during this work, kept because a benchmark that reports only its wins is not evidence.

1. 01

   An entire run scored against a canned template

   A Pydantic serialisation bug made 106 of 106 calls silently return the same fallback text. A full run was scored against it and produced two confident conclusions. Both were withdrawn. A fallback-detection guard now fails any run containing template text.

2. 02

   “83% of misses are ranking, not vocabulary” — withdrawn

   That figure counted any shared token between ticket and path, including `src` (present in 84% of files) and `app` (36%). Recomputed against discriminating tokens only, the picture changed completely and the claim did not survive.

3. 03

   A clone race silently disabled content search

   A concurrent-clone race turned off content search for 11 cases without raising an error. That run was discarded rather than reported with a caveat.

4. 04

   Ranker heuristics carried over from an unrelated product

   Keyword-to-path rules (`hero`, `landing`, `dnd`, `pca`) from a different codebase were awarding fixed points against Directus and Grafana. Deleted. Removing them cost no recall, which is its own finding.

5. 05

   A claim made on the strength of a single run

   Run-to-run variance was assumed small rather than measured — until a claim had already been made on one run. Measuring it showed a five-case swing on 34 identical cases. Differences inside that band are now reported as unchanged, not as small wins.

6. 06

   A judge concern raised, then retracted

   One judge’s ON-vs-NEUTRAL delta was flagged as narrowing against the other’s. Re-running the same judge three times showed its own passes span +0.04 to +0.18 — the second judge sat inside that spread. There was nothing to explain, and judge deltas have been reported as ranges since.