Qwen3.8-27B-DFlash2-W4A16-longctx

Qwen3.8-27B-DFlash2-W4A16 — long-context config

[!IMPORTANT] The weights in this repo are byte-identical to syvai/Qwen3.8-27B-DFlash2-W4A16. Exactly one field in config.json differs:

- "max_position_embeddings": 262144
+ "max_position_embeddings": 393216

That is the whole change. If you are serving at or below 262,144 tokens, use the original — this repo gives you nothing. All quantization credit belongs to syv-ai, and all model credit to inco.ai / z-lab and Qwen.

Why it exists

The DFlash2 drafter ships max_position_embeddings: 262144 with plain, non-YaRN rope. When the target is YaRN-extended past its native 262,144 context, the drafter is asked to draft at positions its rope table does not cover. One token past the limit:

Assertion `index out of bounds: 0 <= ... < 262144` failed
  -> torch.AcceleratorError: CUDA error: device-side assert triggered
  -> vllm.v1.engine.exceptions.EngineDeadError

This kills the entire vLLM engine, not just the offending request — a mid-session hard failure on any long-context deployment.

Why raising it is safe

The drafter uses sliding-window attention (window 2048). RoPE affects attention through relative offsets, and those never exceed the window, so extending the absolute position range does not change what the drafter computes inside any window. Nothing is retrained and no weight is touched; the rope cache is simply built large enough to index.

It is not free, though — beyond the drafter's trained range it is extrapolating, and acceptance degrades:

Measured greedy, num_speculative_tokens=7, on Swift-Qwen3.8-27b-heretic-W8A8-DFlash2, one CMP 170HX (64 GB), fp8_e4m3 KV. At 300k the drafter still beats the in-checkpoint MTP head (2.51), so DFlash2 remains the better speculation choice even there.

Set max_position_embeddings to your served length; 393216 here matches a 1.5× YaRN extension of Qwen3.8-27B's native context.

Also requires a patched vLLM

Being pack-quantized, this drafter additionally needs the quantized-drafter fix — stock vLLM 0.29.0 reads qkv_proj.weight, which a packed checkpoint does not have, and fails at engine init with AttributeError: 'QKVParallelLinear' object has no attribute 'weight' (qwen3_dflash.py:472). See vllm-project/vllm#51684 (comprehensive, upstream) or akumaburn/vllm-dflash2 (narrow stopgap). A BF16 drafter needs no patch.

Usage

vllm serve akumaburn/Swift-Qwen3.8-27b-heretic-W8A8-DFlash2 \
  --max-model-len 393216 --kv-cache-dtype fp8_e4m3 --async-scheduling \
  --hf-overrides '{"text_config":{"rope_parameters":{"rope_type":"yarn","factor":1.5,"original_max_position_embeddings":262144}}}' \
  --speculative-config '{"method":"dflash","model":"akumaburn/Qwen3.8-27B-DFlash2-W4A16-longctx","num_speculative_tokens":7}'

num_speculative_tokens=7 is the architectural maximum (block_size is 8). The target must be unrotated — against a QuaRot-rotated checkpoint this drafter rejects every draft (acceptance exactly 1.00).