Qwen3.8-27B-GGUF

Qwen3.8-27B — GGUF (Q6_K, 8-bit attention projections) with vision & native-MTP speculative decoding

Quantized GGUF conversion of Qwen/Qwen3.8-27B (dense 27B hybrid: 48 Gated-DeltaNet linear-attention layers + 16 full-attention layers, native vision-language, 262K context), tuned and validated end-to-end on an Intel Arc Pro B70 via llama.cpp SYCL.

Files

Conversion notes: converted from the bf16 release; tokenizer and chat template byte-identical to upstream; sha256 receipts of every input shard were taken before conversion. The MTP head in the -MTP file was grafted from the official standalone release EigenLabs/Qwen3.8-27B-MTP-bf16 (15 bf16 tensors → blk.64.* / nextn.* q8_0, GGUF qwen35.nextn_predict_layers = 1); the head's final norm is stored as nextn.shared_head_norm. sha256 Qwen3.8-27B-Q6_K-QKV8-MTP.gguf: 0e9a210bd50b2c057add16bb2a7388a3ac8d9c47de040b8b6c397a1417ec28fb; Qwen3.8-27B-Q6_K-QKV8.gguf: 6bd5285874c123c3a70d7517af1ae075d1d3bcb3e1da7ca03302434a3df90bb4; Qwen3.8-27B-Q6_K-OQ8-MTP.gguf: 2e5c352028a6751c92964bdd4efdfb8f93d4db6cb602508ca2328f4aa6eb4c80.

Quality (measured, not vibes)

KL-divergence against the full bf16 model, 512-token windows × 8, three corpora (wikitext-2 prose, real code, real agent transcripts). Lower is better; "tail" is the 99.9th-percentile per-token KLD (the rare bad token that breaks a program); "same-top" is greedy-token agreement with bf16.

Same-top differences of a few tenths of a point are inside the instrument's noise (~8K tokens); the mean/tail reductions are not. The -MTP files carry byte-identical backbone tensors to their plain twins, so quality is identical by construction; speculative decoding is output-preserving (draft tokens are verified against the target and only exact matches are emitted). Greedy outputs are token-identical to CPU-bf16 on spot prompts. Vision verified through llama.cpp's mtmd.

Why 8-bit attention: at 96K+ context the k/v/qkv projections feed every attention read of the KV cache, and code/agent tokens are low-entropy — one wrong token is a broken program — so the tail matters more than the mean. The rest of the network stays Q6_K because decode on a 32 GB card is memory-bandwidth-bound (measured to 0.6% of the byte roofline): every byte costs speed and context.

Requirements

• Both files need llama.cpp with Qwen3.5-family (qwen35) architecture support (b10355+).
• The -MTP file additionally needs NextN/MTP layer support for the qwen35 arch (qwen35.nextn_predict_layers, blk.N.nextn.* tensors) and a server exposing --spec-type draft-mtp. If your build lacks these, use the plain Q6_K-OQ8 file.
• The model uses interleaved MRoPE and hybrid recurrent+attention memory — context shifting is unavailable by design (clients should budget max_tokens; long-chat prefix reuse works via context checkpoints).

Example (llama-server, vision + native-MTP speculative decoding)

llama-server -m Qwen3.8-27B-Q6_K-QKV8-MTP.gguf --mmproj mmproj-Qwen3.8-27B-F16.gguf \
  -ngl 99 -fa on -c 102400 -np 1 -ub 4096 -b 8192 \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --jinja --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 \
  --reasoning on --reasoning-budget -1

--spec-draft-n-max 3 matches the head's trained block size — larger drafts measured slower (chained acceptance decays past the trained depth). Thinking is on by default (reasoning_effort ∈ xhigh/medium/low via chat_template_kwargs); for non-thinking use {"enable_thinking": false} with the card's non-thinking sampler (temp 0.7, top-p 0.8, presence 1.5). Keep the KV cache at f16 — quantized KV measured a large decode regression at depth on this architecture. On a 32 GB card -c 102400 (+ vision) is the largest context that stays fully resident for the QKV8 file (the OQ8 file fits 106496); past that point llama.cpp silently spills to host memory and prefill, then decode, collapse — measure decode at depth before raising -c.

Measured on Intel Arc Pro B70 (32 GB, llama.cpp SYCL, these exact files)

Serial (no speculation — both files measure identically):

With native-MTP speculative decoding (-MTP file, production server, greedy, 256-token gens):

The gains grow with depth because each accepted draft amortizes the full weight+KV read. Warm TTFT (64-token prompt): 0.59 s. Decode power ~168 W (bandwidth-bound). Numbers are B70-specific; other backends will differ. All numbers were produced with a receipted gate discipline (golden-token equivalence, KLD thresholds, paired A/B/A benches).

Provenance

Converted, quantized, and validated 2026-08-14; native-MTP graft added and gated 2026-08-15; QKV8 files added 2026-08-17 (llama-quantize Q6_K --tensor-type 'attn_k\.weight=q8_0' --tensor-type 'attn_v\.weight=q8_0' --tensor-type 'attn_qkv\.weight=q8_0', from the bf16 release). OQ8 files: Q6_K --output-tensor-type q8_0. No imatrix (at 6-bit the measured quality margin made it unnecessary). Every quality number above was re-measured on the same instrument after a SYCL-backend decode-path fix (a stale activation memo) that had previously made any non-Q6_K weights decode incorrectly on the tuning rig — the published files were validated with that fix in place. Uploaded by Frosty40.