Skip to content

Research and writing

How we think about evaluation, in public.

The answer leakage study, a public LoRA adapter on Hugging Face and the founder's opinion pieces and articles on AI evaluation, education and public policy.

Study

When Execution Feedback Reveals the Expected Output: A Forensic Study of Answer Leakage in Agentic Code Repair

Zenodo, author preprint, not peer reviewed. DOI 10.5281/zenodo.21286873 · 10 July 2026

Coding agents are often given test-runner feedback and then credited with improving a repair. The preprint audits three experimental designs from the author's own research programme and finds that the feedback often exposed the expected output itself. Feedback that exposes a visible test assertion cannot be separated from verification, so any claim that verification helps needs pass-or-fail-only feedback, one shared pool of candidates for every selector, inference clustered by task and immutable hashes linking each transcript to its result row. That is how NASCIA thinks about evaluation: an evaluation has to check what its own feedback gives away.

156 of 169

consumed feedback messages in the primary QuixBugs arm showed the expected output (92.3 per cent)

Open models

A LoRA adapter on the founder's Hugging Face account, which he describes as built for British English prompt optimisation on Mistral 7B. Everything else here comes from the repository itself.

  • Hugging Face

    british-optimizer-mistral-final

    A LoRA adapter for mistralai/Mistral-7B-v0.1, built with PEFT and tagged lora, sft and trl, created in August 2025 with the adapter weights and tokeniser files. The card carries the base model, library and tags; a full model card has not been published yet.

    View on Hugging Face

Opinion pieces and articles

Published in the Brazilian and international press, in the language of each outlet. Titles are kept as published; links open the outlet's own page.

Bring us the process that keeps breaking.

One conversation is usually enough to tell whether an agent belongs there, what it should be measured against, and what it would take to keep it running.