Downloads · 30 days
94
29% of all-time downloads
davetha/Qwen3.8-27B-ABLITERATED-W8A8-gdnint8
Qwen3.8-27B-ABLITERATED-W8A8-gdnint8 is a machine learning model from davetha. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for vllm. The card lists the license as apache-2.0.
INT8 W8A8 quantization of Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16, itself an abliterated derivative of Qwen/Qwen3.8-27B. Abliterated models have had refusal behaviour deliberately removed; that property is inherite…
Downloads · 30 days
94
29% of all-time downloads
All-time downloads
328
Public
Parameters
27.8B
60.4 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors30 GB · 100%
How the weights are stored.
I825.6B · 92%
From the Hugging Face model README
INT8 W8A8 quantization of Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16,
itself an abliterated derivative of Qwen/Qwen3.8-27B. Abliterated models have had
refusal behaviour deliberately removed; that property is inherited here unchanged.
Produced and measured on a single AMD MI210 (gfx90a / CDNA2).
Two things are quantized beyond a normal W8A8 build of this model:
1. The three large gated-delta-net (GDN) projections — in_proj_qkv, in_proj_z,
out_proj — across all 48 linear_attention layers. That is a deliberate departure
from the usual recipe for this architecture, which protects the whole linear-attention
path as quantization-sensitive. Read the evaluation section; a plain W8A8 build is the
conservative choice.
2. lm_head (248,320 x 5,120). This one is close to free — see below.
Measured on one MI210, decode:
| W8A8 baseline | + GDN int8 | + lm_head int8 | |
|---|---|---|---|
| MTP N=1 | 40.5 tok/s | 50.0 | 57.2 |
| per forward step | 43.68 ms | 35.68 | 32.10 ms |
| with dflash spec (N=12) | — | — | 81.3 tok/s |
| KV cache capacity | 298,381 tok | 374,711 | 374,711 (+26%) |
| on disk | 35 GB | 30 GB | 28 GB |
The GDN projections were 5.54 B parameters sitting in BF16 — 11.07 GB read every decode
step against 19.3 GB for everything already quantized, i.e. ~30% of kernel time in one
unquantized tensor family. lm_head was another 2.54 GB in a single call.
Everything the parent W8A8 recipe quantizes (MLPs on all 64 layers, self_attn
projections on the 16 full_attention layers), plus:
| added here | count | |
|---|---|---|
linear_attn.in_proj_qkv | 48 | 10240x5120 |
linear_attn.in_proj_z | 48 | 6144x5120 |
linear_attn.out_proj | 48 | 5120x6144 |
lm_head | 1 | 248320x5120 |
Still BF16, deliberately:
| protected | why |
|---|---|
linear_attn.in_proj_a, linear_attn.in_proj_b | 48x5120 each — 0.42% of the block's bytes, but they feed the recurrent gating (a → softplus → exp → decay, b → sigmoid → beta). Error there compounds along the sequence for no meaningful saving. |
linear_attn.conv1d, all norm | small, sensitive |
re:.*visual.* | vision tower |
mtp.* | MTP draft head, kept BF16 for speculative decoding |
lm_head, embed_tokens |
Symmetric per-output-channel INT8 weights, matching the parent checkpoint exactly:
scale = max|W| / 127 # computed in fp32, then STORED AS BF16
q = round(W / scale.to(bf16)).clamp(-128, 127)
Casting the scale to BF16 before dividing is load-bearing — it is what reproduces the
parent's statistics (row maxima usually 127 but often 120–126, and -128 present).
Dividing by an fp32 scale and casting afterwards yields subtly different weights.
Activations are unchanged: 8-bit per-token dynamic, computed at inference.
lm_headWorth writing down, because it is not obvious and cost several attempts. vLLM's
compressed-tensors path does support a quantized lm_head — ParallelLMHead gets a
CompressedTensorsLinearMethod whenever get_scheme matches, and the embedding()
requirement does not apply to it. The obstacle is purely target matching:
targets: ["Linear"] never matches, because the module's class is ParallelLMHead."lm_head" is still not enough on this model: it is a
ForConditionalGeneration wrapper, so the prefix is nested and only an exact
layer-name match resolves.So this config uses targets: ["Linear", "re:.*lm_head$"] and drops lm_head from
ignore. No vLLM code change is required. (If you are debugging this yourself: testing
find_matched_target with a torch.nn.Linear is misleading — nested names appear to match
via the Linear class-name target. Pass a real ParallelLMHead.)
No calibration data is used or needed — this step is a pure weight transform.
quantize_gdn_int8.py in this repo reproduces it from the parent checkpoint exactly.
Max relative weight error introduced: 0.39%.
Measured against the same checkpoint with GDN projections left in BF16, same harness, same hardware.
The GDN step costs ~2.3% perplexity; the lm_head step costs ~0.01%.
GDN projections, token-weighted over 4,437 tokens: +2.3%
| corpus | tokens | W8A8 | + GDN int8 |
|---|---|---|---|
| prose | 317 | 4.6624 | 4.3646 |
| source A | 2,347 | 2.3573 | 2.4441 |
| source B | 1,773 | 1.9029 | 1.9409 |
lm_head on top of that, token-weighted over 4,463 tokens: +0.012% (two of three
corpora improved — this is noise)
| corpus | tokens | GDN int8 | + lm_head int8 |
|---|---|---|---|
| prose | 317 | 4.3184 | 4.3482 |
| source A | 2,373 | 2.3882 | 2.3879 |
| source B | 1,773 | 1.9488 | 1.9471 |
That asymmetry is expected: lm_head gets a separate scale for each of 248,320 vocab
rows, and it sits outside the recurrent state so its error cannot compound along the
sequence.
Long-context recall: 9/9, identical to the unquantized-GDN control
| prompt tokens | depths 10/50/90% | W8A8 | this |
|---|---|---|---|
| 14,147 | 3/3 | 3/3 | |
| 56,788 | 3/3 | 3/3 | |
| 107,337 | 3/3 | 3/3 |
The needle test was run against both checkpoints on purpose: this hardware declines the fast paged-attention kernel, so a long-context failure could have been the attention path rather than the weights. Running both makes the result attributable.
The needle test above covers the GDN change. It was not re-run for lm_head, because
the mechanism that motivated it — error compounding through a recurrent state — does not
apply to a final projection.
What was NOT evaluated. No standard benchmark suite (MMLU, GSM8K, HumanEval, etc). No multi-turn or long-generation testing. No coding or math evaluation. The perplexity probe is three passages, two of them Python source from one project. The needle test is a single fact. Given that this deliberately quantizes a path the usual recipe protects, and that GDN is recurrent so errors can compound in ways short probes do not reveal, treat these numbers as a smoke test, not a quality claim. If you evaluate it properly, please open a discussion with what you find.
vLLM on ROCm, single MI210:
vllm serve <this-model> \
--tensor-parallel-size 1 \
--max-model-len 131072 \
--gpu-memory-utilization 0.90 \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--speculative-config '{"method": "dflash", "model": <dflash-draft>, "num_speculative_tokens": 12}'
with VLLM_ROCM_USE_AITER=1. Notes for gfx90a specifically:
VLLM_ROCM_USE_AITER=1 needs an image whose AITER carries gfx90a code objects, or
the INT8 GEMM silently falls back to a generic Triton kernel at roughly a third of the
rate. Confirm with grep 'Selected.*ScaledMMLinearKernel' in the server log — you want
AiterInt8ScaledMMLinearKernel, not TritonInt8ScaledMMLinearKernel.--device=/dev/dri/renderD<N>, not all of /dev/dri). vLLM resolves the
architecture once at import from amdsmi's physical device 0 and ignores
HIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES, so a foreign card can switch every gfx9
path off silently. That alone was worth 2.4x here.cudagraph_mode: FULL_DECODE_ONLY matters — vLLM otherwise drops to PIECEWISE under
speculative decoding.Tooling for gfx90a: davetha/mi210-vllm.
Qwen/Qwen3.8-27B → Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 (abliteration) →
W8A8 INT8 (llm-compressor, weights RTN + dynamic per-token activations) → this
(GDN projections and lm_head additionally INT8 — quantize_gdn_int8.py,
quantize_lmhead_int8.py).