Qwen3.8-27B-Opus-Distill-v2-GGUF

Qwen3.8-27B-Opus-Distill-v2-GGUF

GGUF quantizations of barozp/Qwen3.8-27B-Opus-Distill-v2 -- a bugfix release of Qwen3.8-27B-Opus-Distill ("v1").

Why this release exists

A community member (zxbc2023) reported a clean, deterministic bug on v1: certain stacked output-format constraints (e.g. "no prose" + "no markdown") sent the model into a non-converging self-verification loop, burning the entire token budget with zero visible output. Root cause traced to part of v1's training data being sourced from a dataset with reconstructed (not verbatim) reasoning traces. This release retrains on a rebuilt dataset -- every row traced to a verified genuine Opus source, every row scanned (30-way parallel LLM-judge pass over the full set) and dropped if it showed the same pathological pattern.

Bug fix validation

Re-ran the reporter's exact repro (temp=0, seed=1234, fib(n) with stacked constraints):

All three converge quickly, </think> closes normally -- no loop.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (--limit 500). Same protocol as v1, so the delta column is the meaningful signal.

General knowledge (MMLU) and language modeling (wikitext) are essentially unaffected -- in the same noise band as v1. Reasoning (ARC, GPQA) shows a large, real gain over base, not comparable to Qwen's own published GPQA number (thinking-mode-on, different harness) -- this is a same-protocol base-vs-distill delta.

Available quantizations

K-quants (Q8_0-Q3_K_M) need no imatrix. IQ-quants (IQ3_XXS and below) require one to run at all in current llama.cpp.

Which one to pick:

• Best quality with headroom -> Q6_K or Q8_0
• Best quality/size balance -> Q4_K_M (default recommendation)
• 24 GB card -> Q4_K_M; 16 GB card -> Q3_K_M (partial offload)
• Below that -> IQ quants, accept the quality hit

Imatrix

imatrix.dat in this repo is reused from barozp/Qwen3.8-27B-Opus-Distill-GGUF (v1) -- same base architecture, so no need to recompute. As with v1, the native MTP head (blk.64, the nextn.* decoder layer) is never exercised by a normal forward pass, so it has no imatrix coverage; llama-quantize pins that block to q4_K instead of failing.

Vision (mmproj)

The vision tower is in Qwen3.8-27B-Opus-Distill-v2-mmproj-f16.gguf (~0.9 GB). Load it alongside any quant for image/video input:

llama-server -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-v2-mmproj-f16.gguf

Text-only usage does not need mmproj and runs fine without it.

Quick start

# text-only chat
llama-cli -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf -no-cnv

# multimodal server
llama-server -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-v2-mmproj-f16.gguf

# with self-speculative decoding (compute-bound hardware)
llama-cli -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf -no-cnv --spec-type draft-mtp -fa on

Training details (source safetensors model)

• Base: Qwen/Qwen3.8-27B -- dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
• Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
• Data: barozp/opus-reasoning-distill-v2 (11,716 examples, 100% verified-genuine Opus traces)
• Run: 1 epoch (696 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~4h15m on A100 80GB
• Final validation loss: 0.4334
• Vision + MTP: carried over byte-for-byte from the base checkpoint -- never trained

Source chain

The full Qwen3.8-27B Opus Distill family:

• Qwen/Qwen3.8-27B — base model
• barozp/Qwen3.8-27B-Opus-Distill-v2-GGUF — v2, GGUF quants
• barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 — v2, FP8 build
• barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit — v2, MLX 4-bit
• barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-8bit — v2, MLX 8-bit
• barozp/Qwen3.8-27B-Opus-Distill-v2 — v2, BF16 weights
• barozp/Qwen3.8-27B-Opus-Distill-GGUF — v1, GGUF quants
• barozp/Qwen3.8-27B-Opus-Distill — v1, BF16 weights

This release: v2, GGUF quants (the card you are reading).

Thanks to zxbc2023 for the bug report that made this release possible.

Related

• barozp/Qwen3.8-27B-Opus-Distill-v2 -- BF16 source model, full card.
• barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 -- block-FP8 safetensors build for sglang / vLLM / transformers; includes paired BF16-vs-FP8 benchmark data.
• barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit and MLX-8bit -- Apple Silicon MLX builds via mlx-vlm.