Swift-Qwen3.8-27b-W4A16-AutoRound-MTP-BF16

Swift-Qwen3.8-27B W4A16 (AutoRound, BF16 MTP head)

A community 4-bit weight-only quantization of ukisai/Swift-Qwen3.8-27b, UkisAI's reasoning-efficient fine-tune of Qwen3.8-27B. It is not an official UkisAI release.

• W4A16: int4 weights (group size 128, symmetric), 16-bit activations, made with Intel AutoRound 0.15.1 and exported as compressed-tensors. vLLM picks the int4 kernels up automatically (Machete on Hopper, Marlin elsewhere).
• 19.47 GB on disk versus about 56 GB for the BF16 parent.
• BF16 MTP head: the multi-token-prediction module is kept at full precision, so self-speculative decoding works as it does on the parent.

Benchmarks

Run on the three benchmarks of the parent page's Quantized models table, under that table's protocol. The parent columns are copied from its model card; this repo's column was measured here.

• Hit token cap: samples that reached the benchmark's generation cap while still reasoning, so they produced no final answer and score as incorrect. All 24 AIME misses of this kind are cap hits; with the 250,000-token cap the BF16 parent scores 94.00% instead of 84.00%, so most of the AIME gap reflects the 32,768-token protocol.

• Swift BF16 (parent): the parent's main BF16 table (5 seeds). Its AIME figure uses a 250,000-token cap and is not comparable to the 32,768-token protocol here.

• Swift quant (parent): the parent's own mixed-precision W4A16 quantization of Swift, from its Quantized models table.

• GPQA reports thinking tokens; IFBench and AIME report completion tokens, as on the parent page. For reference, the BF16 parent averaged 8,855 GPQA thinking tokens and 4,657 IFBench completion tokens under its main-table protocol.

• IFBench: 2 of 1,200 samples were still generating when the run hit its time budget and are excluded; they cannot move the score by more than 0.1 points.

• GPQA-Diamond: per-seed 88.38, 88.89, 89.39, 87.37, 87.88 (std 0.8); 990 of 990 samples scored.

• IFBench: per-seed 70.33, 71.33, 72.24, 68.90 (std 1.43); loose prompt-level 78.88%; 1198 of 1200 samples scored.

• AIME 2026: per-seed 83.33, 86.67, 80.00, 83.33, 86.67 (std 2.79); 150 of 150 samples scored.

<details> <summary><strong>Evaluation settings</strong></summary>

• Serving: vLLM 0.27.1 (the parent's version), --reasoning-parser qwen3, context 262,144, single NVIDIA H200 on Modal, MTP off (speculative decoding does not change the output distribution; it was left off because it lowered batch throughput).
• Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 0, repetition_penalty 1, reasoning_effort xhigh (the chat template default), request seed = sample index.
• GPQA-Diamond: official Idavidrein/gpqa diamond split (198 questions), choices shuffled per question and seed, answer read from a final Answer: X line.
• IFBench: allenai/IFBench_test (300 prompts), scored on the final answer (reasoning excluded) with the official IFBench checker at commit 1c40f0c; prompt-level strict accuracy.
• AIME 2026: MathArena/aime_2026 (30 problems), answer taken from the last \boxed{}; a generation that hits the 32,768-token cap has no final answer and scores as incorrect.
• The parent did not publish its prompts or harness, so absolute numbers can differ slightly for harness reasons as well as quantization. The parent's other six benchmarks were not rerun; see the parent card.

</details>

Quantization details

The layer selection follows dbirks/Qwen3.8-27B-W4A16-AutoRound, except that the MTP head stays BF16 and is listed in quantization_config.ignore so vLLM builds the draft module unquantized.

How to use

vLLM

vllm serve causal/Swift-Qwen3.8-27b-W4A16-AutoRound-MTP-BF16 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --port 8000

On one H200 (0.92 GPU memory utilization) vLLM 0.27.1 reports a KV cache of 1,704,711 tokens, enough for about 6.50x full-length 262K requests. Measured throughput under a saturated long-reasoning load was about 3,706 output tokens/s (231 concurrent requests).

Optional MTP decoding

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

With MTP on, the same load showed a 54% draft acceptance rate (1.63 accepted tokens per step). MTP speeds up individual requests at low concurrency; at high batch sizes it lowered total throughput in our test.

Sampling

Use the parent's recommended settings: temperature 1.0, top_p 0.95, top_k 20, min_p 0.

License and access

This repository is a quantized derivative of ukisai/Swift-Qwen3.8-27b and carries the same terms. UkisAI's contribution is licensed under the Swift Open License v1.0; the underlying Qwen3.8-27B is licensed under Apache 2.0. See NOTICE and the parent card for details, including the revenue threshold above which commercial use needs a Swift Enterprise License from UkisAI.

Citation

@misc{swift-qwen3.8-27b,
  title  = {Swift-Qwen3.8-27B},
  author = {UkisAI},
  year   = {2026},
  url    = {https://huggingface.co/ukisai/Swift-Qwen3.8-27b}
}

Acknowledgements

Swift was trained by UkisAI with access to 8x NVIDIA H100 GPUs from the NVIDIA Innovation Lab. Quantization for this repository ran on a Modal L40S and evaluation on a Modal H200.