Fine-Tuning Qwen3-4B on Agent Traces: the Thinking-Budget Failure Benchmarks Don't Catch

This post documents the v2 release of Parable-Qwen3-4B, a QLoRA fine-tune of Qwen3-4B on 10.6k real agent traces. The recipe is standard for this class of fine-tune: completion-only loss masking, a 30% general-instruction replay mix, checkpoint gating on a held-out benchmark. The result worth writing up is not the benchmark table. It is a failure mode in hybrid-thinking models that none of the standard evals surface, and that affected 21% of prompts in v1.

The setup

Qwen3-4B is a hybrid-thinking model: by default it reasons inside a <think> block before emitting an answer. Fine-tuning on agent traces (recorded multi-step sessions of a frontier model using tools) transfers agentic behavior into the 4B — planning, tool use, terminal habits. Training used TRL with DataCollatorForCompletionOnlyLM, LoRA r=16 on all projection matrices, two seeds, on free-tier GPUs (Kaggle P100, max_seq_length 2048).

One recipe detail matters for what follows: the corpus applies the Qwen chat template with empty think blocks, so the fine-tune learns to answer directly instead of reasoning aloud.

The failure benchmarks miss

On a 34-prompt review suite of ordinary coding and terminal tasks, v1 of this fine-tune returned an empty response on 7 of 34 prompts — it spent its entire 2,600-token budget inside the think block and never emitted an answer. A 21% silent-failure rate, invisible in every metric tracked during training:

• HumanEval doesn't see it — scoring harnesses extract code from whatever text is returned; an empty response is one failed problem, indistinguishable from a wrong answer.
• Loss curves don't see it — thinking tokens are well-predicted tokens. The loss on a response that never terminates in an answer looks healthy.

The same trap affects benchmarking the base model: in default thinking mode, Qwen3-4B burns its budget reasoning and scores near zero on a chat-format HumanEval harness. Any comparison of a non-thinking fine-tune against a default-mode hybrid-thinking base is rigged in the fine-tune's favor. The fix, on llama.cpp's OpenAI-compatible server:

# llama-server ... --jinja
client.chat.completions.create(
    model="m", temperature=0.0, max_tokens=3500,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    messages=[{"role": "user", "content": prompt}],
)

Results

Q4_K_M builds, greedy decoding, thinking disabled on every row, HumanEval+ scored with EvalPlus:

Coding is parity with v1 (+1.2 HumanEval, −1.2 HumanEval+ — noise). The v2 change that matters is the first row: every prompt gets an answer. Against the base, the fine-tune costs 3.6 HumanEval points — about a third of the regression the identical recipe produces on Granite-3B, which suggests the forgetting cost of agent-trace SFT is strongly base-dependent.

Across the family

Held-out trace loss for the same recipe across three bases (lower is better, base vs fine-tune):

Takeaways for fine-tuning Qwen3

• Benchmark the base model under your exact harness before training — hybrid-thinking bases need enable_thinking: false or they score near zero for the wrong reason.
• Keep a small suite of ordinary prompts and read the raw outputs. v1's worst defect was invisible in aggregates and obvious in thirty seconds of reading.
• If your corpus uses empty think blocks, your fine-tune becomes mode-invariant — v2 scores identically with thinking on or off, evidence the SFT reshaped generation rather than adding surface style.

Weights (GGUF Q4–Q8 + safetensors), both training seeds, and all eval outputs: Hugging Face · Ollama · ModelScope. Full recipe and the rest of the series: the Parable page.