A forensic audit showing how often test feedback handed coding agents the expected answer

The problem

A coding agent that repairs a function is usually run in a loop: propose a patch, run a test, read the message that comes back, try again. When the loop is given feedback from a test runner and the repair rate rises, the feedback is credited with the improvement. The question the founder's research programme set out to test was narrower: does reliability depend on who owns the verdict that stops the loop, the model itself or a supervisor whose verdict cannot be forged?

That question cannot be answered while the feedback message carries the answer. A traceback that prints assert f(x) == y does more than report a failure. It discloses the relation the visible test checks and the literal target value. Any gain measured after such a message is confounded with a task specific hint, and the study calls that a measurement problem.

What was built

The preprint "When Execution Feedback Reveals the Expected Output: A Forensic Study of Answer Leakage in Agentic Code Repair" was published on Zenodo on 10 July 2026 as an author preprint, version 1.0.0, not peer reviewed, under a Creative Commons Attribution 4.0 licence. Its author is NASCIA's founder, writing under his University of São Paulo affiliation. It is a forensic study of his own research programme, not a product, and it finds no validated clean positive evidence for the benefit it had been designed to show.

Three experimental designs were audited, each with its own benchmark and forensic status: ten author designed tasks with Qwen2.5-Coder 14B; HumanEvalFix with Qwen2.5-Coder 1.5B over 80 tasks and three seeds; QuixBugs with the same 1.5B model over 29 tasks and three seeds, giving 87 observations per arm. The paper contributes two leakage audits specific to those harnesses, a provenance bound diagnostic and a control protocol for future studies.

The audits are simple to state. For QuixBugs, a keyword scan for the word "expected" in every consumed feedback message, followed by a strict recount that requires the harness's own template, "Input gave X, expected Y". For HumanEvalFix, a matcher that takes each assertion line from the official example test and requires the complete task specific line to appear in the feedback, with a round admitted only if the SHA-256 of the transcript's final user message matches the hash stored in the result row. Rounds that cannot be bound are counted as clean, so every HumanEvalFix fraction reported is a lower bound.

The paper replaces an earlier preprint from the same programme. That earlier work was withdrawn in full after a forensic audit found that its outcomes had not been produced by a live model loop, and none of its numbers enter the new analysis. The record names the withdrawn preprint on its own landing page, and the paper states why: disclosing the failure is part of the result.

How it works

The diagram shows the loop under study and the audit built around it. A language model proposes a patch; a visible test runs it; a feedback message goes back to the model. The audit reads the returned strings themselves and binds each message to its result row by hash; beside the real verifier, three control arms of the same loop hand the model a wrong, a stale or a shuffled message instead of the real feedback.

The QuixBugs campaign had seven pre-registered arms: a blind baseline, in-prompt feedback, self-stop, out-of-band verification with rich feedback, and the three integrity controls. The invalidation rule was fixed before the first campaign attempt: the design is invalid if any control beats the baseline by at least 0.07 with a paired McNemar exact p below 0.05. Only after the primary campaign had closed and the rule had fired were two exploratory arms specified, each time stamped before its own execution: a scaffold only loop with content free messages, and an out-of-band verifier whose message says only pass or fail.

Inference is clustered by task. The 87 observations per arm are 29 tasks repeated over three seeds, so the task is the unit for the two sided sign test and the paired bootstrap. Wilson intervals describe cell rates, and the McNemar test is kept wherever it decided a registered rule. No equivalence margin was pre-registered, so a large p value is never read as proof of no effect.

Evidence

In the primary 1.5B QuixBugs arm, 156 of 169 consumed feedback messages, 92.3 per cent, rendered the fixed expected output field (Zenodo record description and paper section 6, fact answer-leakage-study-08). In HumanEvalFix, at least 316 of 417 out-of-band feedback rounds, 75.8 per cent, exposed the task specific rendered assertion, and 289 of 417, 69.3 per cent, exposed an explicit equality against an expected value; both are lower bounds (paper Table 4, page 8, fact answer-leakage-study-09). The earliest design emitted raw pytest output that was never scanned, so it remains unresolved (paper section 8, fact answer-leakage-study-10).

The pre-registered invalidation rule fired. A nominally inert stale control beat the blind baseline by 0.138, with a paired McNemar exact p of 0.0018; the post campaign analysis, honest about clustering, corroborated the decision with a task level sign test p of 0.0117 and a paired task clustered bootstrap interval of [0.046, 0.241] (Zenodo record description and paper section 6, page 7, fact answer-leakage-study-11). The primary causal reading of the campaign is therefore void.

The exploratory arms show what remains once the leak is removed. Rich feedback out-of-band resolved 33 of 87 task seed cells; the pass or fail only verifier resolved 21 of 87; the scaffold only loop resolved 21 of 87; the blind baseline resolved 22 of 87. The rich versus pass or fail contrast was +0.138, with a sign test p of 0.0020 and a bootstrap interval of [0.069, 0.230]. The two clean arms showed no detectable gain over baseline at the achieved precision, and the paper is explicit that this is not an equivalence result (Zenodo record description and paper Tables 2 and 3, page 7, fact answer-leakage-study-12).

The stored QuixBugs run is forensically recomputable: across 870 rows the self audit found zero first call model mismatches, and zero of 870 row outcomes diverged when the stored solutions were re-executed in a fresh subprocess; the 2,875 repair producing model calls were not rerun (paper section 6, page 8, and section 11, page 12, fact answer-leakage-study-18). The HumanEvalFix corpus is not: a later 3B run overwrote 783 row referenced transcript files, so its exact historical leakage rates are unrecoverable and are reported as conservative lower bounds (Zenodo record description and paper section 7, fact answer-leakage-study-13).

The limits are stated in the paper itself. The experiments cover function level Python repair and one model family, mainly the 1.5B model served through a mutable Ollama tag; final resolution on QuixBugs is about 95 per cent collinear with passing the one visible case; the clean arms were specified after the campaign; and the negative finding is a falsification of the programme's own earlier evidence, not a universal claim that leak free verification is ineffective (paper section 9 and section 13, fact answer-leakage-study-19). The study used public benchmarks and locally served models, with no human participants and no personal data (paper section 12, fact answer-leakage-study-26).

Stack

The harnesses ran locally: Qwen2.5-Coder models served through Ollama, Python test execution in an isolated subprocess under a CPU cap, and pytest tracebacks or the harness's own result template as the feedback channel. The analysis package is standard library Python that regenerates the arm table and re-executes every stored solution; the repository carries 3,389 SHA-256 manifest entries (paper section 11, fact answer-leakage-study-24). The evidence repository remains private, and the record says it must not be described as a public, complete reproducibility artefact (fact answer-leakage-study-21).

What it demonstrates

Define. The unit, the primary contrast, the arm family and the rule that would invalidate the design were written down before the data. When the rule fired, the headline result was dropped rather than explained away.

Instrument. The evaluation did not stop at the scoreboard. The strings returned to the model were audited, the audit was bound to the raw rows by hash, and integrity controls ran alongside the real verifier. This is the discipline behind NASCIA's second step: build the harness first, so that an improvement can be told from a leaked answer.

Integrate. Provenance is treated as part of the estimand. Every row names its run and model, stores the exact request and response, and binds each consumed message by SHA-256, so a later overwrite is detectable and its damage can be bounded rather than guessed.

Hand over. The paper closes with a six point minimum protocol for any future study: pass or fail as the default feedback, one immutable candidate pool for every selector, matched budgets, immutable identifiers, hash binding verified before aggregation, and a fresh recomputation before any causal narrative. That is the same shape as the runbook NASCIA leaves with a client: the tests, the limits and the words, all written down.