Swift-Qwen3.8-27b-heretic-W8A8-DFlash2

Swift-Qwen3.8-27b-heretic — GPTQ W8A8 INT8 (unrotated) · DFlash2-ready

[!CAUTION] Research artifact. Safety alignment has been deliberately removed. This model will attempt to comply with harmful, dangerous, illegal and unethical requests that the base model refuses, with no content moderation. See Safety. Provided for research purposes only, with no warranty and no liability accepted by the author — see the disclaimer at the bottom.

An INT8 W8A8 quantization of akumaburn/Swift-Qwen3.8-27b-heretic built without the QuaRot rotation, so it can be served with the DFlash2 speculative drafter while keeping the native CUTLASS INT8 throughput of an 8-bit checkpoint.

• Weights: INT8, per-channel symmetric, GPTQ learned rounding.
• Activations: INT8, per-token dynamic → compressed-tensors int-quantized, native CUTLASS INT8 tensor-core path.
• Left unquantized: lm_head, embed_tokens, the vision tower, GatedDeltaNet in_proj_a/in_proj_b recurrent gates, the MTP head, all norms.
• No rotation, no SmoothQuant — plain GPTQ. 30 GB.

[!IMPORTANT] Why unrotated. The SmoothQuant+QuaRot sibling puts the residual stream in a rotated basis. DFlash2's fc was trained on unrotated target hidden states, so against a rotated checkpoint it rejects every draft — measured acceptance length exactly 1.00, while every other gate (output quality, KL, throughput) still looks fine. Dropping the rotation restores drafting (4.14 on code, 3.11 on prose) at a cost of 2.9× quantization error under real serving numerics (served KL 0.0337 vs 0.0117).

Variants

served KL = KL(source ‖ build) under vLLM (real serving numerics), WikiText-2, top-512, 12,264 token positions. first-token KL = full-vocabulary at the first generated token over 100 harmless instruction prompts, under transformers.

[!NOTE] The two metrics disagree in an informative way. On served natural text the 4-bit and unrotated-8-bit builds are effectively tied (0.0354 vs 0.0337); the 4-bit penalty is concentrated at high-entropy decision points, where first-token KL puts it 2.7× worse. Conversely the rotation is worth 2.9× on the served metric but only 1.7× first-token — expected, since rotation exists to tame activation outliers for INT8 activation quantization, which only the served measurement exercises.

Performance

vllm bench serve, one CMP 170HX (64 GB), TP=1, fp8_e4m3 KV, async scheduling, random 1,024-token inputs. Prefill = c × input_len ÷ mean TTFT (instantaneous rate during prefill, not end-to-end input throughput).

[!TIP] Match the speculative depth to your concurrency. DFlash2 k=7 gives up to 2.8× decode at c=1 and still wins at c=8, but at c≥16 it costs throughput — at c=32 it cuts prefill from 18,942 to 2,768. This is speculative overhead on an already-saturated GPU (rejected drafts are wasted compute), not a property of the quantization: with speculation off, this checkpoint reaches 18,942 tok/s prefill, matching the rotated sibling.

Rule of thumb: DFlash2 for c ≤ 8, disable it for c ≥ 16.

Full DFlash2 sweep at 1,024-token inputs (k=7): prefill 2,878 / 4,134 / 6,209 / 7,755 / 5,400 / 2,768 and decode 110.7 / 159.8 / 232.7 / 379.1 / 369.5 / 396.9 for c = 1 / 2 / 4 / 8 / 16 / 32; acceptance length stays in the 3.3–4.2 band throughout (that sweep used randomized inputs; see Speculative acceptance for the per-domain figures).

Speculative acceptance

Greedy (temperature 0), k=7, concurrency 1, 48 prompts per set, DFlash2 drafter, identical harness for both checkpoints:

Acceptance is governed by prompt domain, not by the checkpoint or by the abliteration. Structured code drafts far better than short open-ended prose (−32 % accepted tokens per draft, significant). A benign-prose control scores 3.267 against 3.206 for abliterated-content prose — a −2.7 % difference with a 95 % CI of ±0.16, i.e. not significant: abliteration does not degrade speculation.

The 4-bit sibling's acceptance edge over this build is small — +6.5 % on code (marginally significant) and not significant on prose. Its advantage is wall-clock decode (+36 % at c=1), traded against this build's much faster prefill and lower quantization error.

The drafter's block_size is 8, so k=7 is the architectural maximum.

[!NOTE] An earlier version of this card reported 3.36 / 33.8 %, measured on a 4-prompt file, alongside 5.12 for the 4-bit sibling. Both came from that too-small sample and overstated the gap between the builds; the 48-prompt figures above supersede them.

Quantization fidelity

KL(source ‖ build), measured two ways:

Dropping the rotation costs 2.9× on the served metric and 1.7× first-token. The served figure is the one to weigh: it is measured under vLLM with real INT8 activations, which is precisely what the rotation exists to protect. The first-token measurement runs in transformers with fp16 activations and so understates the rotation's value — the earlier "~1.7× cost" figure on this card was based on that metric alone and understated the trade.

That is the price of DFlash2 compatibility, and it is a real one. If you do not need the speculative drafter, the rotated sibling is meaningfully more faithful at identical size and speed.

Against the 4-bit build this checkpoint is ~2.7× closer on first-token KL but essentially tied on served KL (0.0337 vs 0.0354). Its advantages over W4A16 are prefill throughput (native CUTLASS INT8 vs Marlin dequant, ~2× at short prompts) and high-entropy fidelity — not general output quality on ordinary text. It costs 11 GB more VRAM for that.

Refusal behaviour is inherited unchanged from the source: 0/100 hard refusals, with the same ~20–24 % chain-of-thought deflection band described on the source card.

Serving

Choosing a drafter

The drafter is a separate checkpoint from this model. Two work, and which one you can use depends on your vLLM build:

Stock vLLM 0.29.0 already ships DFlash2, but _build_fused_kv_buffers reads qkv_proj.weight directly. A pack-quantized (W4A16) drafter has no such attribute, so it dies at startup with

AttributeError: 'QKVParallelLinear' object has no attribute 'weight'
  vllm/model_executor/models/qwen3_dflash.py:472

A BF16 drafter takes the dense fast path and needs no patch at all — that is the simplest way to run this model. The one-function fix for the quantized drafter is at github.com/akumaburn/vllm-dflash2.

[!CAUTION] Serving past 262,144 tokens kills the engine. Both drafters declare max_position_embeddings: 262144. At a longer position the drafter's rope table is indexed out of bounds and the CUDA assert index out of bounds: 0 <= ... < 262144 takes down the whole vLLM engine (EngineDeadError), not just the request. This is reachable whenever the target is YaRN-extended past its native 262,144 context.

Fix: raise max_position_embeddings in the drafter's config.json to your served length, or use the ready-made akumaburn/Qwen3.8-27B-DFlash2-W4A16-longctx (identical weights, that one field set to 393216). The weights are untouched, and it is safe because the drafter uses sliding-window (2048) attention — only relative positions inside the window matter. Measured with the enlarged table: acceptance 3.82 at 150k and 2.70 at 300k (degraded by extrapolation beyond the drafter's trained range, but working). Unfixed, 300k is a hard crash.

Measured on identical hardware and harness, the quantized drafter is worth +4 % decode and 2.4 GB of VRAM; acceptance between the two is statistically identical. At long context the VRAM is the bigger prize, since it becomes KV cache.

# single user / low concurrency
vllm serve akumaburn/Swift-Qwen3.8-27b-heretic-W8A8-DFlash2 \
  --tensor-parallel-size 1 --max-model-len 32768 \
  --kv-cache-dtype fp8_e4m3 --async-scheduling \
  --speculative-config '{"method":"dflash","model":"<path>/Qwen3.8-27B-DFlash2-W4A16","num_speculative_tokens":7}'

# batch serving (c >= 16): drop the speculative config entirely

All quantized matmul dimensions are multiples of 128 (no INT8 padding).

Safety

Refusal behaviour has been deliberately removed; this model produces content the source declines, including dangerous, illegal or unethical material, with no moderation. Intended for interpretability/safety research, red-teaming and evaluation by people who understand and accept those risks. Do not deploy it where it can reach people who have not consented to unfiltered output. You are responsible for your use and for compliance with all applicable laws.

Disclaimer

This model is provided for research purposes only, "AS IS", without warranty of any kind, express or implied. The author accepts no liability for any use of this model or any consequences arising from it. By downloading or using it, you accept sole responsibility for your use and for compliance with all applicable laws and regulations.

Licences: the Swift contribution is under the Swift Open License v1.0 (© UkisAI); the underlying Qwen3.8-27B is Apache-2.0 (© Alibaba Cloud). Both licence files are included. Abliteration via Heretic; quantization via llm-compressor.