Swift-Qwen3.8-27b-heretic-W4A16

Swift-Qwen3.8-27b-heretic — GPTQ W4A16 (INT4) · DFlash2-ready

[!CAUTION] Research artifact. Safety alignment has been deliberately removed. This model will attempt to comply with harmful, dangerous, illegal and unethical requests that the base model refuses, with no content moderation. See Safety. Provided for research purposes only, with no warranty and no liability accepted by the author — see the disclaimer at the bottom.

A 4-bit W4A16 quantization of akumaburn/Swift-Qwen3.8-27b-heretic, built for single-user interactive decoding with the DFlash2 speculative drafter on Ampere-class (SM 8.0) GPUs.

• Weights: INT4, group 128, symmetric, GPTQ learned rounding, no act-order (so the Marlin kernel needs no g_idx) → compressed-tensors pack-quantized.
• Activations: 16-bit (bf16). Weight-only quantization.
• Left unquantized: lm_head, embed_tokens, the vision tower, GatedDeltaNet in_proj_a/in_proj_b recurrent gates, the MTP head, all norms.
• 19 GB (vs 30 GB for the W8A8 sibling, 52 GB for the BF16 source).

[!IMPORTANT] This checkpoint is deliberately not rotated. The W8A8 sibling applies an offline QuaRot rotation, which puts the residual stream in a rotated basis and makes the DFlash2 drafter — whose fc was trained on unrotated hidden states — reject every draft (measured: acceptance length exactly 1.00). Skipping rotation is what makes this build usable with DFlash2. SmoothQuant is likewise omitted: it migrates quantization difficulty into the weights, which is the wrong trade when only weights are quantized.

Variants

served KL = KL(source ‖ build) under vLLM (real serving numerics), WikiText-2, top-512, 12,264 token positions. first-token KL = full-vocabulary at the first generated token over 100 harmless instruction prompts, under transformers.

[!NOTE] The two metrics disagree in an informative way. On served natural text the 4-bit and unrotated-8-bit builds are effectively tied (0.0354 vs 0.0337); the 4-bit penalty is concentrated at high-entropy decision points, where first-token KL puts it 2.7× worse. Conversely the rotation is worth 2.9× on the served metric but only 1.7× first-token — expected, since rotation exists to tame activation outliers for INT8 activation quantization, which only the served measurement exercises.

DFlash2 column is acceptance length on harmless48 (48 code/prose tasks, greedy, k=7; 1.00 = every draft rejected). Acceptance is governed by prompt domain, not by the build: the same checkpoints score ~3.2 on conversational prose. See Speculative acceptance below.

[!TIP] Match speculative depth to concurrency. Measured on the unrotated W8A8 (same model and dataset, only the speculative config varying, 1,024-token inputs): DFlash2 k=7 gives 2.8× decode at c=1 and still wins at c=8, but at c≥16 it costs throughput — at c=32 prefill falls from 18,942 to 2,768 tok/s. That is speculative overhead on a saturated GPU, not a property of any checkpoint. Use DFlash2 for c ≤ 8; disable it for c ≥ 16.

Choosing between this and the 8-bit siblings

These are different operating points, not better/worse. Measured on one CMP 170HX (64 GB), TP=1, fp8_e4m3 KV, async scheduling, identical datasets. Two workloads are shown because the answer depends on prompt length — quoting one number for "prefill" is misleading.

Long prompts (~16k in, 16 out):

Short prompts (1,024 in, 256 out) — prefill = c × in_len ÷ mean TTFT:

• Interactive / single user (c ≤ 4): this build — up to 1.5× the decode throughput of the 8-bit builds at 1k-token inputs, and 11 GB of freed VRAM.
• Anything larger, or prefill-heavy: an 8-bit sibling. Prefill here is a flat ~0.55× of W8A8 (Marlin int4→bf16 dequant vs native CUTLASS INT8), and decode saturates around 255 tok/s where W8A8 climbs past 500.
• If you want DFlash2 and 8-bit fidelity: the unrotated W8A8 — same speculative capability, 2.7× closer to the source on first-token KL, though statistically tied with this build on served KL (0.0337 vs 0.0354). Choose it for prefill throughput and high-entropy fidelity, not for a blanket quality win.

Performance

vllm bench serve, TP=1, fp8_e4m3 KV, async scheduling, DFlash2 drafter with num_speculative_tokens=7.

• Prefill point: distinct ~16k-token realistic prompts (concatenated code/math), 16 output tokens. Throughput is compute-saturated at ~1,750 tok/s for this workload and does not scale with c; TTFT grows linearly as long-prompt requests queue. (On synthetic random-token prompts the same server does scale with c — the saturation is workload-specific, not universal.) The flat ~0.52× versus the W8A8 sibling is the Marlin int4→bf16 dequant path versus native CUTLASS INT8 GEMMs.
• Decode point: realistic code/math prompts (~140 input tokens), 2,048 output cap. Decode saturates above c≈16 (570 → 578) as the model stops being weight-bandwidth-bound; the 8-bit sibling keeps climbing to 840.
• DFlash2 acceptance is healthy and stable (3.7–3.9) across the whole range.

Speculative acceptance

Greedy (temperature 0), k=7, concurrency 1, 48 prompts per set, DFlash2 drafter, identical harness for both checkpoints:

Acceptance is governed by prompt domain, not by the checkpoint or by the abliteration. Structured code — boilerplate, rigid syntax, predictable continuations — drafts far better than short open-ended prose (−32 % accepted tokens per draft, 95 % CI ±0.17, significant). Whether that prose is benign or is exactly the content the model was decensored for makes no measurable difference (−2.7 %, 95 % CI ±0.16 — not significant). Abliteration does not degrade speculation.

Against the 8-bit sibling the acceptance edge is small: +6.5 % on code (95 % CI ±0.18, marginally significant) and not significant on prose. This build's real speculative advantage is wall-clock rather than acceptance: +36 % decode throughput at c=1 on both sets, because 4-bit weights are cheaper to stream.

The drafter's block_size is 8, so k=7 is the architectural maximum — there is no headroom from drafting deeper.

[!NOTE] An earlier version of this card reported 5.12 / 58.9 %, measured on a 4-prompt file. That sample was far too small and optimistic; the 48-prompt figures above supersede it. The 8-bit sibling's previously published 3.36 was measured on that same 4-prompt file and is likewise superseded by 4.138 — the two builds are much closer than those numbers implied.

Quantization fidelity

KL(source ‖ this), measured two ways — they disagree, and the disagreement is the most useful thing on this card:

On the first-token metric 4-bit weights cost ~4.7× the divergence of the rotated 8-bit build — and it exceeds the abliteration's own divergence from the unmodified Swift model (0.0763), meaning quantization perturbs this checkpoint more than the uncensoring did.

On the served metric that gap almost vanishes: 0.0354 here versus 0.0337 for the unrotated 8-bit build at the same activation precision story — a 5 % difference, not a 2.7× one. Averaged over 12,264 positions of ordinary text the two are effectively equivalent.

Both numbers are real; they measure different things. Natural-language continuation is dominated by low-entropy, highly predictable positions where 4-bit weights are adequate. The first-token metric samples exactly the opposite regime — the high-entropy moment where an instruction's response is chosen among many plausible openings — and there the coarser weight grid costs real accuracy. Expect this build to feel close to 8-bit on flowing prose and continuation, and to diverge most at decision points: the first token of a reply, branch points in reasoning, structured-format choices.

The rotated W8A8 leads on both metrics, and by more on served (2.9×) than first-token (1.7×) — consistent with the rotation's actual job, which is taming activation outliers for INT8 activation quantization. Only the served measurement exercises INT8 activations at all; the transformers first-token measurement runs them in fp16 and therefore understates what rotation buys.

Refusal behaviour is inherited unchanged from the source: 0/100 hard refusals, with the same ~20–24 % chain-of-thought deflection band described on the source card.

Serving

Choosing a drafter

The drafter is a separate checkpoint from this model. Two work, and which one you can use depends on your vLLM build:

Stock vLLM 0.29.0 already ships DFlash2, but _build_fused_kv_buffers reads qkv_proj.weight directly. A pack-quantized (W4A16) drafter has no such attribute, so it dies at startup with

AttributeError: 'QKVParallelLinear' object has no attribute 'weight'
  vllm/model_executor/models/qwen3_dflash.py:472

A BF16 drafter takes the dense fast path and needs no patch at all — that is the simplest way to run this model. The one-function fix for the quantized drafter is at github.com/akumaburn/vllm-dflash2.

[!CAUTION] Serving past 262,144 tokens kills the engine. Both drafters declare max_position_embeddings: 262144. At a longer position the drafter's rope table is indexed out of bounds and the CUDA assert index out of bounds: 0 <= ... < 262144 takes down the whole vLLM engine (EngineDeadError), not just the request. This is reachable whenever the target is YaRN-extended past its native 262,144 context.

Fix: raise max_position_embeddings in the drafter's config.json to your served length, or use the ready-made akumaburn/Qwen3.8-27B-DFlash2-W4A16-longctx (identical weights, that one field set to 393216). The weights are untouched, and it is safe because the drafter uses sliding-window (2048) attention — only relative positions inside the window matter. Measured with the enlarged table: acceptance 3.82 at 150k and 2.70 at 300k (degraded by extrapolation beyond the drafter's trained range, but working). Unfixed, 300k is a hard crash.

Measured on identical hardware and harness, the quantized drafter is worth +4 % decode and 2.4 GB of VRAM; acceptance between the two is statistically identical. At long context the VRAM is the bigger prize, since it becomes KV cache.

vllm serve akumaburn/Swift-Qwen3.8-27b-heretic-W4A16 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8_e4m3 \
  --async-scheduling \
  --speculative-config '{"method":"dflash","model":"<path-to>/Qwen3.8-27B-DFlash2-W4A16","num_speculative_tokens":7}'

Without a drafter it serves as a plain W4A16 checkpoint on the Marlin kernel. Optional: VLLM_MARLIN_INPUT_DTYPE=int8 makes Marlin consume INT8 activations on these same weights (an effective "W4A8" without a second checkpoint); on patched builds VLLM_MARLIN_INT8_INCLUDE_RE=mlp applies it selectively. Neither was used for the numbers above.

All quantized matmul dimensions are multiples of 128 (no INT4 padding).

Safety

Refusal behaviour has been deliberately removed; this model produces content the source declines, including dangerous, illegal or unethical material, with no moderation. Intended for interpretability/safety research, red-teaming and evaluation by people who understand and accept those risks. Do not deploy it where it can reach people who have not consented to unfiltered output. You are responsible for your use and for compliance with all applicable laws.

Disclaimer

This model is provided for research purposes only, "AS IS", without warranty of any kind, express or implied. The author accepts no liability for any use of this model or any consequences arising from it. By downloading or using it, you accept sole responsibility for your use and for compliance with all applicable laws and regulations.

Licences: the Swift contribution is licensed under the Swift Open License v1.0 (© UkisAI); the underlying Qwen3.8-27B is Apache-2.0 (© Alibaba Cloud). Both licence files are included. Abliteration via Heretic; quantization via llm-compressor / compressed-tensors.