Qwen3.8-27B-Unleashed-GGUF

Qwen 3.8 27B UNLEASHED — GGUF

<div> <a href="https://huggingface.co/JonathanColetti/Qwen3.8-27B-Uncensored"><img src="https://img.shields.io/badge/base-Uncensored%20bf16-a78bfa?style=flat-square" alt="base"></a> <a href="https://unsloth.ai/docs/basics/dynamic-3.0-ggufs"><img src="https://img.shields.io/badge/recipe-Unsloth%20Dynamic%203.0-22d3ee?style=flat-square" alt="recipe"></a> <img src="https://img.shields.io/badge/context-262144-0ea5e9?style=flat-square" alt="context"> <img src="https://img.shields.io/badge/license-Apache%202.0-64748b?style=flat-square" alt="license"> </div>

✅ All 9 tiers verified and live (2026-08-21)

UD-IQ1_M and UD-IQ2_S were briefly published broken, then withdrawn, and are now rebuilt, load-tested, and back. If you downloaded either before 2026-08-21 22:00 UTC, re-download. No other tier was ever affected.

If you are quantizing this model yourself, this is the part worth reading. Upstream builds low-bit tiers with a different architecture, not just different bit widths:

	Q3_K_XL and above	IQ1_M / IQ2_S
tensors	506	498
highest block	64 (MTP head at q6_k)	63 (MTP pruned)
block_count	65	64
nextn_predict_layers	1	0

Their imatrix has 496 entries because it matches that pruned topology. Quantizing a 65-block f16 against it fails with Missing importance matrix for tensor blk.64.attn_k.weight.

--prune-layers 64 alone is not enough — it removes the tensors but leaves the model declaring an MTP head, so loading dies on blk.63.nextn.eh_proj.weight not found: a structurally valid file containing an unloadable model. You also have to rewrite the metadata:

llama-quantize --imatrix <imatrix> --tensor-type-file IQ1_M.txt \
  --prune-layers 64 \
  --override-kv qwen35.block_count=int:64 \
  --override-kv qwen35.nextn_predict_layers=int:0 \
  model-f16.gguf out.gguf IQ1_M 12

Every file here is now gated on exit code → magic bytes → an actual load test before upload. Header checks alone let two broken builds through; only loading the model catches it.

Uncensored Qwen3.8-27B, quantized with per-tensor dynamic bit allocation.

Most uncensored GGUFs are uniform quants — every layer gets the same treatment. These use a per-tensor type map, so sensitive tensors keep precision while tolerant ones compress hard. The result: a Q3 that outperforms a Q4 3.3 GB larger, with full 250k-token retrieval.

Provenance, stated plainly. The per-tensor quant map was extracted from unsloth/Qwen3.8-27B-GGUF (Unsloth Dynamic 3.0) by reading the tensor-type table from their published files, then applied to JonathanColetti/Qwen3.8-27B-Uncensored weights with that repo's imatrix. The recipe is Unsloth's work, not ours — we did not independently derive it. All upstream projects are Apache 2.0. See Credits.

---

Measurements

Every number below is from one machine, one harness, same session — RTX 4090 24 GB, llama.cpp with DFlash2 speculative decoding, q4_0 KV cache, -c 262144, wikitext-2 perplexity at 60 chunks.

⚠️ Do not compare these PPL values against numbers from other setups. Chunk count and harness change the absolute scale; only within-table comparisons are meaningful.

Tiers marked (pending) are uploaded and usable; their full speed/retrieval sweep is still running.

What the table says:

• The dynamic Q3 beats every Q4 tested, including a Q4 2.2 GB larger.
• Against uniform Q3 on identical weights: −0.13 PPL and +33% throughput for 0.6 GB more.
• Quality matches Unsloth's own Q3 within error (Δ 0.0043, σ ≈ 0.045) — call it a tie, with the difference being uncensored weights at ~1 GB more.
• Q2 is the floor. The dynamic recipe halves the damage of a naive Q2 but still costs ~3.8%. Q3 is where quality holds.

Speed: what you should actually expect

Measured on real traffic (n=1,696 requests, RTX 4090, Q3_K_XL, dflash2 draft). Throughput depends far more on how many tokens you generate than on context depth, because every request pays fixed overhead (prefill, sampler init, draft warm-up) that only amortises over a long generation:

Peak observed on a single request: ~195 tok/s. Do not plan around peak.

If you are running an agent that mostly emits short tool calls, expect the ~25 tok/s end of that table, not the top. That is not a defect of this quant - it is the fixed cost of a 27B model per request, and it applies to any GGUF of this model. Speculative decoding needs a few tokens to build acceptance momentum, so very short replies finish before the draft warms up.

Lowering context does not fix this. Only 17 of 65 layers are full-attention (the rest are linear/DeltaNet), so KV work is a small share of per-token cost. We measured 61 tok/s median at 16-64k context versus 30 tok/s at <4k - deeper context was faster, because those requests generated more tokens. Context depth is not the lever; generation length is.

Retrieval at depth

Exact-string needle recovered from a 250,806-token context (98% window occupancy):

---

Files

All nine tiers, same recipe, same weights, plus a tested vision projector. Sizes are on-disk (decimal GB).

VRAM math for a 24 GB card: weights + ~5 GB KV (262k @ q4_0) + ~2 GB draft model + compute buffers. The KV cache is small for a 27B because Qwen3.8 is hybrid DeltaNet — only 17 of 65 layers are full attention.

Quick start

llama-server -m Qwen3.8-27B-Unleashed-UD-Q3_K_XL.gguf \
  -ngl 999 -c 262144 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --jinja

Charts

Which file do I want?

---

How the base model ranks

External context for the base model (stock Qwen3.8-27B, bf16 — not our quants):

• Artificial Analysis (Aug 2026): ranks alongside DeepSeek V4 and GPT-5.6-class models on the Intelligence Index — discussion · charts
• "Beats all medium models (40B–150B); same score as DeepSeek V4 Flash" — HN summary of AA data
• Hybrid DeltaNet architecture: only 17 of 65 layers are full attention → KV cache at 262k ctx is ~5 GB (q4_0), which is why full context fits on a 24 GB card at all

Our quants inherit these capabilities minus quantization loss (measured above).

Optimal settings — these matter

Which tier should you actually use?

Every tier was measured on the same box (RTX 4090, llama.cpp-dflash2) on the same day: wiki.test perplexity, generation speed, MMLU (10 questions x 57 subjects = 570, greedy, thinking disabled), needle-in-haystack retrieval, and a 20-prompt refusal battery.

MMLU here is n=120 per tier (subject-stratified) for ladder comparison; the headline 82.98% figure below is the full n=570 run on Q3_K_XL. n=120 carries roughly ±4pp, so treat small gaps as noise.

Three findings worth stating plainly

0. UD-IQ1_M scores at random chance. Do not use it. MMLU is 4-way multiple choice, so 25% is the floor for guessing. IQ1_M scored 25.83% — statistically indistinguishable from noise — with perplexity 32% worse than Q3 and a failed needle retrieval at 32k. It is the fastest file here (77 tok/s) and it retains no usable knowledge. We are shipping it with the numbers rather than quietly omitting the tier: if you have an 8 GB card, run UD-IQ2_S instead — it is 1.7 GB larger, holds 70.83% MMLU, and passes retrieval.

The degradation is not gradual. IQ2_S matches Q2_K_XL exactly (70.83%) while being 1.4 GB smaller; one step further down falls off a cliff. The usable floor of this model is 2-bit.

1. Perplexity is not capability. UD-IQ4_XS has the best perplexity in the entire ladder (6.3502) and is not the best model — it scores below UD-Q3_K_XL on MMLU while being 1.1 GB larger and 5% slower. If you are choosing a quant, choose on task benchmarks, not PPL.

2. Nothing above Q3 buys anything. UD-Q6_K is 8.9 GB larger and 28% slower than UD-Q3_K_XL, and does not score better. Quality rises steeply from Q2 to Q3 and then flattens. The recommended tier is the knee of every curve, not a compromise.

3. Abliteration survives quantization. Refusal rate is 0.0% at every tier, down to 9.9 GB. Whatever the low-bit tiers cost you, it is not censorship creeping back in.

Capability: MMLU

Measured on this exact quant (UD-Q3_K_XL), not the parent bf16 — so it includes quantization loss and abliteration loss.

Protocol: greedy (temperature 0), thinking disabled, single-letter answer extraction, 10 questions per subject across all 57 MMLU subjects, 0 parse failures.

Reading this honestly: ~2.3 points below stock bf16, and that gap covers both the abliteration and 3-bit quantization. For a 13.2 GB file that also keeps 262k context and runs at 25-56 tok/s on one consumer GPU (see the speed section - it depends heavily on how many tokens you generate), that is the trade being offered. If you need every last point of benchmark accuracy and don't need uncensored, run the stock bf16.

Refusal behaviour

20-prompt boundary battery (security mechanics, drug pharmacology, graphic fiction, social engineering, morally repugnant character writing, suggestive romance). Scored by refusal-marker detection on the response opening; every prompt produced a substantive answer (806-1004 chars on the last three).

Does the abliteration method matter?

We tested this directly. Taking a different uncensored base — OBLITERATUS V2, which uses complementary blending (SVD + LEACE) rather than the Heretic method — and running it through the identical UD-Q3_K_XL recipe, same hardware, same day, so the only variable was the base weights:

Identical scores. The two models disagree on 44 individual questions (92.3% prediction agreement) — they are genuinely different models that happen to land in the same place.

At 3-bit, the abliteration technique appears to be roughly fungible: different methods, same measured capability, same refusal behaviour. Worth knowing before you switch bases chasing a benchmark claim.

Caveat, stated plainly: n=20 is a spot check, not a 800-prompt corpus. The parent model's card describes refusals as substantially reduced, not eliminated - you will likely find edge cases. This measures our quant of it, at n=20, on the prompts listed in refusal-results.jsonl.

MTP and vision

• MTP head: present. The nextn.* multi-token-prediction tensors survived abliteration (re-grafted by the parent repo) and are retained in every quant here (eh_proj at Q6_K).
• Vision: bundled and tested. mmproj-Unleashed-f16.gguf (0.93 GB) is in this repo, built from the same parent weights as these quants. Every other uncensored GGUF of this model we are aware of is text-only, because convert_hf_to_gguf.py defaults to text and drops the 333 vision tensors unless you pass --mmproj.

# vision is one extra flag - the weights themselves are unchanged
llama-server \
  -m Qwen3.8-27B-Unleashed-UD-Q3_K_XL.gguf \
  --mmproj mmproj-Unleashed-f16.gguf \
  -ngl 999 -c 262144 -fa on \
  --cache-type-k q4_0 --cache-type-v q4_0 --jinja

Then post images to /v1/chat/completions with an image_url content part (OpenAI format), or use llama-mtmd-cli.

What we verified (Q3_K_XL + this projector, single 4090):

Tested with synthetic images generated at test time, so ground truth is exact and the model cannot be scored generously. Abliteration did not break cross-modal alignment - that was the open risk and it came back clean.

Cost: the projector adds ~0.9 GB resident VRAM. Text-only requests never touch the vision encoder, so there is no generation-speed penalty for having it loaded; you pay a one-time prefill per image. Omit --mmproj and behaviour is identical to a text-only build.

Not yet measured by us: image-heavy benchmark accuracy, OCR quality, multi-image prompts.

Running it

Recommended llama.cpp server config — this is the setup all measurements were taken with:

llama-server \
  -m Qwen3.8-27B-Unleashed-UD-Q3_K_XL.gguf \
  -ngl 999 -c 262144 -b 2048 -ub 512 -fa on \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --cache-reuse 256 \
  --jinja --reasoning-effort medium \
  --temperature 1 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0

Speculative decoding with DFlash2 (optional, big win)

Every speed number on this card was measured with a DFlash2 draft model attached. It is not part of these files - it is a separate ~2 GB model that predicts several tokens ahead, which the main model then verifies in one pass. On our runs it accepted 48.6% of drafted tokens with a mean accepted run of 3.92 tokens. Note that acceptance on real agent traffic is lower (median ~0.37-0.51) than on benchmark prompts.

Get the draft model from spiritbuun/Qwen3.8-27B-DFlash2 (Q8_0, ~2 GB) and add:

  -md Qwen3.8-27B-DFlash2-Q8_0.gguf -ngld 999 \
  --spec-type draft-dflash --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
  --spec-draft-type-k q4_0 --spec-draft-type-v q4_0

Tuning note: we A/B'd the draft parameters. n_max=6, p_min=0.75 beat both n_max=10, p_min=0.55 (41% acceptance) and n_max=10, p_min=0.40 (30% acceptance). Higher p_min wins here - drafting more aggressively costs more than it gains. Without a draft model everything still works, just slower; subtract roughly a third of the throughput.

Notes:

• --cache-reuse 256 matters more than it looks. With a stable system prompt, throughput at 120k depth measured higher than at 32k — prefix reuse amortizes the prefill.
• q4_0 KV cache is what makes 262k fit in 24 GB. At f16 KV you will OOM well before that.
• Sampler values are Qwen3.8's published recommendations, not tuned by us.

---

Method

1. Read the tensor-type table from each published Unsloth UD GGUF header (HTTP range request — metadata only, no full download).
2. Emit a --tensor-type-file map, one tensor=type line per quantized tensor (506 for Q3_K_XL).
3. Run llama-quantize against jonathan's uncensored f16 with his published imatrix and that map.

The map is non-obvious and worth looking at. Across every tier from IQ1_S to Q8_K_XL, the ~96 ssm_alpha / ssm_beta tensors are held at Q8_0 — even in the 1-bit builds. Those gate Qwen3.8's DeltaNet path. Our own hand-rolled attempt (uniform Q3_K on those tensors, heavier attention) scored 6.5865 — the worst result we measured. The specific per-layer allocation is the contribution, and it is Unsloth's.

Limitations

• Uncensored ≠ unaligned. Refusal behaviour is substantially reduced, not eliminated. See jonathan's card for measured refusal rates.
• PPL is a weak proxy for task quality. A 16-task BenchLoop coding suite (local-first LLM benchmark harness) scored identically (14/16) across every quant tested here — it did not discriminate at this tier. Treat PPL as a relative signal.
• Retrieval was tested with exact-string needles, which is the easy case. Multi-hop reasoning at 250k was not measured.
• Single-machine numbers. No cross-hardware validation.
• Vision: use the mmproj from the upstream repo; not included here.

Credits

• JonathanColetti — the uncensored base weights (abliteration via Heretic, MTP head re-grafted and verified) and the imatrix used for these quants. None of this exists without that work.
• Unsloth — the Dynamic 3.0 per-tensor quantization recipe. We extracted and reapplied their map; the method and the analysis behind it are entirely theirs. If you can use the censored base, use their files directly.
• Qwen — the base model.
• llama.cpp — quantization and inference.

Apache 2.0, inherited from all upstream components. Unsloth and Qwen names are used descriptively for attribution only; no affiliation or endorsement implied.