Agent-Trace Fine-Tuning of Small Language Models under Constrained Compute

Naively fine-tuning small language models on agent traces regresses general coding ability by roughly 19 points, and the regression is invisible unless you measure for it. We show that a recipe combining completion masking, a replay mix, and dev-gated checkpoint selection recovers about half of the lost capability. We name and eliminate “session leakage”, a failure mode in which transcript artifacts from the training corpus surface in model outputs. Along the way, three of our headline results turned out to be artifacts of the evaluation apparatus itself. We argue that at this scale, evaluation discipline matters more than compute.