The First Nanbeige4.2-3B Fine-Tune: Looped Layers, a Retention Cliff, and a Run That Crossed Three GPUs

Nanbeige4.2-3B was released on 2026-07-24. Five days later we shipped Parable-Nanbeige4.2-3B, a QLoRA fine-tune on real agent traces and, as far as we can find, the first published fine-tune of this base anywhere. Being first on a base nobody has touched is mostly a distribution decision, but the technical route there produced three things worth writing down: a silent half-depth failure mode unique to looped-layer models, a sequence-length cliff that would have discarded three quarters of the training data, and a checkpoint-relay pattern that let one training run move between different GPUs without losing progress.

A 22-layer model that is 44 layers deep

Nanbeige4.2-3B runs its 22 transformer layers twice per forward pass: num_loops: 2 in the config, effective depth 44 in a 4.17B-parameter footprint. The trap is in the GGUF conversion path: llama.cpp's converter reads num_loops from the config and silently defaults it to 1 if the field is missing. A merge or save step that rebuilds config.json without the key ships a model that loads cleanly, generates fluently, and runs at exactly half its trained depth. No gate downstream catches that, so our pipeline asserts num_loops == 2 three times: after the LoRA merge, in the converter log, and in the final GGUF header.

The retention cliff

Agent traces are long. Fine-tuning with completion-only masking means a training row only contributes loss if its assistant span survives truncation, so the max sequence length silently decides how much of the corpus exists at all. We measured retention across candidate lengths on 400 real traces before committing GPU time:

That is not a curve, it is a cliff: between 2,048 and 3,072 tokens retention jumps from 24% to 97%. Train at 2,048 and three quarters of the corpus is silently discarded before the first step. The uncomfortable part is that 2,048 was the "safe" choice on a 16GB P100. The measured activation cost was ~2.6GB per extra 1,024 tokens over a 7GB floor (doubled compute from the layer loop included), which put 3,072 at 14.8GB. It fit, with 1.2GB to spare, and the peak measured in the final run was 14.8GB exactly.

One run, portable across GPUs

The run was designed to survive interruption from the start: every 50 optimizer steps the full checkpoint (adapter, optimizer, scheduler and RNG state) is relayed to a private Hugging Face repo, and the training script resumes from the highest relayed checkpoint it finds. That makes the run portable across heterogeneous hardware, and we used that deliberately: early steps ran on a P100 at ~130s/step for these 3,072-token sequences, and the run finished on an A10G at ~28s/step, 4.6x faster per step. One practical note from moving a checkpoint between environments: transformers loads checkpoint RNG state with weights_only=True, which rejects a numpy-2-pickled rng_state.pth on a numpy-1 container. Delete the file and resume; losing data-order RNG is cheaper than losing the run. With a 50-step relay cadence, no interruption (spot preemption, session limit, crash) can ever cost more than 50 steps.

What the fine-tune actually does

Held-out trace loss fell from 2.333 to 2.133: an 8.6% reduction on sessions the model never saw, the smallest delta in the Parable series. That is the honest headline, and it makes sense: Nanbeige4.2 starts closer to the trace distribution than any base we have used. We make no benchmark claims over the base; its own card reports SWE-bench Verified 63.6 (vendor self-reported), this variant trains the reasoning voice on prose answers, and the two statements live in different currencies. The artifact details, quants and usage caveats live on the model page; the weights are on Hugging Face.