Qwen3-4B-High-Level-Critic-SFT-DPO-GGUF

Qwen3-4B High-Level Critic SFT-DPO — GGUF

GGUF conversion and quantizations of code-critic-model/Qwen3-4B-Critic-SFT-DPO, the 4B SFT + DPO code-agent critic from Steer, Don't Solve: Training Small Critic Models for Large Code Agents (arXiv:2606.21811).

The upstream checkpoint is based on Qwen/Qwen3-4B-Instruct-2507 and is trained to supervise coding agents at the trajectory level. Rather than solving the coding task itself, the critic examines the agent's work so far, identifies trajectory-level errors, and provides short corrective guidance.

This repository makes one additional inference-oriented modification: the paper's high-level critic system prompt is embedded directly in the GGUF chat template.

No additional model training was performed.

Why this version exists

The upstream checkpoint contains the trained critic weights, but its standard chat template does not permanently include the high-level supervisor prompt used to elicit the intended critic behavior.

These GGUFs embed that prompt directly into tokenizer.chat_template.

As a result, a llama.cpp-compatible client can send an agent trajectory as an ordinary user message without separately supplying the critic system prompt.

For example, a request containing only:

{
  "messages": [
    {
      "role": "user",
      "content": "<agent trajectory>"
    }
  ]
}

is rendered approximately as:

<|im_start|>system
[embedded high-level critic/supervisor instructions]
<|im_end|>
<|im_start|>user
<agent trajectory>
<|im_end|>
<|im_start|>assistant

If the calling application supplies an additional system message, it is appended after the embedded critic instructions within the system section.

Do not separately inject the same high-level critic prompt when using these GGUFs, or the instructions will be duplicated.

Critic behavior

The embedded prompt instructs the model to monitor an LLM-based coding agent for twelve classes of trajectory-level errors:

Specification errors

1. Task Specification Violations
2. Role Specification Violations
3. Step Repetition
4. Termination Condition Unawareness

Reasoning errors

1. Problem Misidentification
2. Tool Selection Errors
3. Hallucinations
4. Information Processing Failures

Coordination errors

1. Task Derailment
2. Goal Deviation
3. Context Handling Failures
4. Verification Failures

The high-level prompt deliberately tells the critic to be conservative. It should normally allow a competent coding agent to continue working unless there is clear evidence of a problem.

It is also instructed not to solve the coding problem for the agent: no code, shell commands, implementation recipes, or detailed alternative technical approaches.

Upstream model

The DPO checkpoint starts from code-critic-model/Qwen3-4B-Critic-SFT and was further trained using preference pairs of critic responses.

The authors describe it as the strongest 4B critic evaluated in the paper.

The underlying Qwen3-4B-Instruct-2507 architecture has a native context length of 262,144 tokens. The upstream critic authors use a 65,536-token maximum model length in their published serving example.

Available GGUFs

llama.cpp usage

General-purpose Q6_K

llama-server \
  -m Qwen3-4B-High-Level-Critic-SFT-DPO-Q6_K.gguf \
  -c 32768 \
  -ngl 99

Memory-constrained configuration

For a memory-constrained GPU (~6 GB VRAM), Q5_K_M with quantized KV cache can leave substantially more room for the full context:

llama-server \
  -m Qwen3-4B-High-Level-Critic-SFT-DPO-Q5_K_M.gguf \
  -c 32768 \
  -ngl 99 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

If additional VRAM needs to be recovered, reducing the physical microbatch size with -ub can reduce compute-buffer requirements.

For example:

llama-server \
  -m Qwen3-4B-High-Level-Critic-SFT-DPO-Q5_K_M.gguf \
  -c 32768 \
  -ngl 99 \
  -ub 128 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

The 32,768-token context in these examples is a practical deployment choice, not the model's architectural maximum.

Tested configuration

The Q5_K_M GGUF has been tested with llama.cpp at a 32,768-token context using Q8_0 K/V cache.

Example measured memory allocation on an Apple M2 was approximately:

With this configuration, a simple direct critic request used 948 prompt tokens and produced a correctly structured supervisor response without any runtime-supplied system message.

Observed performance on that test was approximately:

• Prompt processing: 207 tokens/sec
• Generation: 18.6 tokens/sec

These numbers are hardware-, backend-, batch-, and llama.cpp-version-dependent and should not be interpreted as general benchmarks.

Intended use

This model is intended to run alongside an LLM-based coding agent.

At intervals during an agent trajectory, the critic receives the work performed so far and returns structured feedback containing:

• detected error categories;
• evidence for detected errors;
• a recovery action;
• an overall task status; and
• brief overall guidance.

License

The upstream checkpoint is released under the Apache License 2.0. This GGUF conversion retains that license.

Credits

All model training, datasets, evaluation, and the critic methodology belong to the original authors of Steer, Don't Solve and the maintainers of code-critic-model.

Qwen3-4B-Instruct-2507 was developed by the Qwen team.

This repository provides only the GGUF conversion, quantizations, and modified inference chat template described above.