Qwen3.8-27B-heretic-ara-DFlash2-fp8

Qwen3.8-27B-heretic-ara-DFlash2 — selective FP8 experiment

This is an experimental, mixed-precision FP8 quantization of alphakek/Qwen3.8-27B-heretic-ara-DFlash2, prepared and validated by magiccodingman.

It is a target-specific speculative-decoding drafter, not a standalone language model. Use it only with heretic-org/Qwen3.8-27B-heretic-ara or a weight-compatible quantization of that exact target, such as the paired magiccodingman/Qwen3.8-27B-heretic-ara-fp8 repository.

Why selective FP8

Quantizing every drafter matrix was measurably less stable. The selected profile quantizes the large MLP projections and preserves DFlash-sensitive paths in BF16.

The following remain BF16 deliberately:

• Q/K/V projections required by the current fused context-KV path
• attention output projections
• target-hidden-state adapter
• dynamic convolution projections and kernels
• candidate-selector projection and codebooks
• norms and other small parameters

Validation

Eight prompts were run through the real BF16 27B target. Identical hidden states from target layers 5, 19, 33, 47, and 61 were supplied to the BF16 and selective- FP8 drafters, producing 56 compared draft positions through the target's actual 248,320-token LM head.

Structural validation, block geometry, finite scales, preserved-tensor hashes, controlled MLP arithmetic, official dflash 0.1.0 loading, and repository checksums passed. See FP8_EXPERIMENT_VALIDATION.md and the included JSON reports for full details.

The selector-path metric is intentionally strict: one early token difference changes the predecessor used by later selector positions. Target verification protects the final output distribution, but proposal drift may still change acceptance length and throughput.

Usage with the paired target

The drafter repository intentionally has no tokenizer or multimodal processor. Those artifacts come from the target model repository and were not present in the upstream drafter repository.

Example using the proposed Hub repository names:

vllm serve magiccodingman/Qwen3.8-27B-heretic-ara-fp8 \
  --tensor-parallel-size 4 \
  --reasoning-parser qwen3 \
  --kv-cache-dtype fp8 \
  --speculative-config '{"model":"magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8","method":"dflash","num_speculative_tokens":6}'

At the time this checkpoint was validated, DFlash2 support for vLLM was still being developed in vllm-project/vllm#52816. Use a compatible DFlash2-enabled build. Backend support for this selective block-FP8 layout must also be verified on the deployment system.

Hardware note: RTX 3090 does not execute native W8A8 FP8. The local tests validated the serialized weights and proposal behavior by dequantizing FP8 matrices to BF16. Native FP8 kernel throughput was not measured.

Performance expectation

The upstream BF16 drafter card reports 123.2 tok/s and 4.24 average acceptance length on its 4× RTX 3090 TP4 test, versus 116.9 tok/s and 4.01 for the stock z-lab drafter. Those are upstream BF16 results, not results for this FP8 derivative.

Before relying on this checkpoint, compare the BF16 and FP8 drafters on the same hardware, prompts, concurrency, speculative-token count, and runtime build. Measure both tokens/second and acceptance length.

Provenance and credits

The derivative chain is:

1. Qwen Team — Qwen/Qwen3.8-27B
2. heretic-org/Qwen3.8-27B-heretic-ara, the target model
3. z-lab/Qwen3.8-27B-DFlash2, the stock target-conditioned DFlash2 drafter
4. alphakek/Qwen3.8-27B-heretic-ara-DFlash2, SpecForge-tuned for the Heretic ARA target and warm-started from the z-lab drafter
5. This selective-FP8 conversion by magiccodingman

DFlash and its implementation are credited to the z-lab/dflash project and its authors. The upstream target's ARA transformation credits p-e-w/heretic, timrohrbaugh/heretic, and the ARA contribution.

This quantization is an independent derivative and is not an official Qwen, Heretic, z-lab, Inco, alphakek, or vLLM release.

Limitations and responsibility

This checkpoint is experimental. It may load correctly yet provide no speedup if the runtime dequantizes its FP8 matrices or lacks compatible fused kernels. Selector-path drift can reduce speculative acceptance even when target verification preserves final-generation correctness. It also inherits the target model's limitations and reduced refusal behavior. Users are responsible for deployment safeguards, legal compliance, and generated content.

License

Apache License 2.0. See LICENSE. All upstream notices and attributions remain applicable.