Qwen3-8B-Critic-SFT

Qwen3-8B-Critic-SFT

The main 8B critic from Steer, Don't Solve: Training Small Critic Models for Large Code Agents.

A critic sits next to a frozen coding agent. Every k agent steps it reads the trajectory so far and returns a short structured critique: which error categories it detects, the evidence, a recovery action, the task status, and one line of overall guidance. It steers the agent; it does not write the patch. This model is Qwen3-8B fine-tuned on critiques written by Claude Opus 4.6 for trajectories of two different agents.

All released models and datasets are listed on the organization page. Code, configs, and launch scripts are in the critic-training repository.

Where it appears in the paper

Original run name: qwen3-8b-full-sft-prm-r2egym-swebench-instructions-k5-cwm-plus-qwen-opus-distill-32k-lr5e6-multiturn. This is the name that appears in the repository's configs, logs, result directories, and the LiteLLM cost registry (there under the shubhamrgandhi/ prefix).

Training data

code-critic-model/critic-sft-cwm-qwen, 6,447 examples.

• Tasks: 500 R2E-Gym instances from the matplotlib, moto, and sympy repositories. These are disjoint from SWE-bench Verified at the instance level.
• Agents that produced the trajectories: CWM-32B (500 trajectories, 4,532 examples) and Qwen3-Next-80B-A3B-Instruct (483 trajectories, 1,915 examples).
• Teacher: Claude Opus 4.6, queried every 5 agent steps with the high-level prompt. The high-level prompt asks for error detection and short guidance and forbids full code solutions. The paper calls this the "high-level" or "concise" prompt.

Training setup

Full-parameter SFT with LLaMA-Factory. The config is finetuning/qwen3_8b_critic_full_sft_l40s_train_multiturn_resumable.yaml in the repository.

Results

Resolve rate on SWE-bench Verified (500 instances), no critic versus this critic. Numbers are from Table 1 of the paper and take the better of k=5 and k=10 for each configuration.

Resolve rate on 300 Multi-SWE-bench instances (Table 2), with k=5. The critic saw only Python trajectories during training.

How to use

The critic was served with vLLM in bf16 and called through the repository's fork of mini-swe-agent, which inserts a critique into the agent's context every k steps.

vllm serve code-critic-model/Qwen3-8B-Critic-SFT \
    --served-model-name Qwen3-8B-Critic-SFT \
    --dtype bfloat16 --max-model-len 65536 --port 8071

Then, from the repository root, run an agent with the step-aware critic prompt at k=5:

bash scripts/run_critic_max150.sh prm_issue_res_instructions_step_aware 5 0 qwen3-80b \
    --prm Qwen3-8B-Critic-SFT --prm-node <vllm-host>:8071 --slice :500 \
    --prefix-dir <path to the matching no-critic run>

The launcher passes the --prm name to LiteLLM, which needs a matching entry in mini-swe-agent/configs/litellm_model_registry.json to price the calls. Copy the block for the original run name to a new key Qwen3-8B-Critic-SFT, or serve under the original run name instead. Without a registry entry the critic call fails and the agent runs without critiques. The full inference procedure, including the agent-side configs and the no-AWS path, is in QUICKSTART.md and HANDOVER.md.

To call the critic directly, reuse a training record as the prompt. The system message and the trajectory encoding are exactly what the model saw during training.

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "code-critic-model/Qwen3-8B-Critic-SFT"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="bfloat16", device_map="auto")

record = load_dataset("code-critic-model/critic-sft-cwm-qwen", split="train")[0]
messages = record["messages"][:-1]   # drop the teacher critique, keep system + trajectory
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False,
                                 return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=1024, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Limitations

The critic was trained on Python repositories only and on trajectories in the mini-swe-agent format (one bash command per step). It has been evaluated as a critic for other agents, not as a stand-alone coder, and its critiques are only as reliable as the teacher's on the training distribution.

Citation

@misc{gandhi2026steerdontsolvetraining,
  title={Steer, Don't Solve: Training Small Critic Models for Large Code Agents},
  author={Shubham Gandhi and Yiqing Xie and Atharva Naik and Ruichen Zhu and Carolyn Rose},
  year={2026},
  eprint={2606.21811},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  url={https://arxiv.org/abs/2606.21811}
}