Skip to content

Research and writing

How we think about evaluation, in public.

A study, two open model repositories and the founder's opinion pieces on AI evaluation, education and public policy, grouped by year.

Study

When Execution Feedback Reveals the Expected Output: A Forensic Study of Answer Leakage in Agentic Code Repair

Zenodo, author preprint, not peer reviewed. DOI 10.5281/zenodo.21286873 · 10 July 2026

Coding agents are often given test-runner feedback and then credited with improving a repair. The preprint audits three experimental designs from the author's own research programme and finds that the feedback frequently exposed the expected output itself: in the primary QuixBugs arm, 156 of 169 consumed feedback messages (92.3 per cent) rendered the fixed expected-output field. The conclusion is that feedback which exposes a visible test assertion is causally confounded with verification, and that any claim of a verification benefit needs pass-or-fail-only feedback, a same-candidate-pool selection control, task-clustered inference and immutable transcript-to-row hashes. It is how NASCIA thinks about evaluation: an evaluation must test what its feedback reveals.

Read on Zenodo

Open models

Two repositories on Hugging Face under the founder's account, which he describes as LoRA adapters for British English prompt optimisation on Mistral 7B. Beyond that, only what the repositories themselves state is stated here.

  • Hugging Face

    british-optimizer-mistral-final

    A LoRA adapter for mistralai/Mistral-7B-v0.1, trained with supervised fine-tuning (TRL and PEFT) and published in August 2025 with the adapter weights and tokenizer files. The card carries the base model, library and training tags; a full model card has not been published yet.

    View on Hugging Face
  • Hugging Face

    british-prompt-optimizer-mistral-lora

    A repository created in August 2025 under the same account. Its model card and files have not been published yet, so nothing is stated about it beyond its name.

    View on Hugging Face

Opinion pieces

Published in the Brazilian and international press, in the language of each outlet. Titles are kept as published; links open the outlet's own page.