ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ

bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ

NVFP4 (W4A4) quantization of bottlecapai/ThinkingCap-Qwen3.8-27B, a Qwen3.8-27B fine-tune, built with llm-compressor and stored in the compressed-tensors format vLLM loads natively. 23.4 GB on disk, against ≈56 GB for the bf16 source.

The MTP (multi-token-prediction) head and the vision tower are kept in bf16 and work: self-speculative decoding and image input both serve from this repo.

What is quantized

Weights were smoothed with AWQ (activation-aware quantization) before rounding. The counts below are the language model's Linear layers; the vision tower and the MTP head are bf16 throughout and are not part of them.

A mixed 4-bit/FP8 layout rather than 4-bit everywhere: the attention and linear-attention projections are where a flat 4-bit quantization of this architecture loses accuracy, and FP8 costs little memory over 4-bit at that share of the weights.

Hardware

Needs Blackwell (sm_100 / sm_120: RTX PRO 6000, B200, GB200). Weights and activations are FP4, which is what reaches the FP4 tensor cores — vLLM serves this through CutlassNvFp4LinearKernel. This build's numbers on this card come from an RTX PRO 6000; on other GPUs serve NVFP4 (4-bit weights, 16-bit activations) instead.

Serving

Tested on vLLM 0.29.0.

vllm serve bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --max-model-len 49152 \
  --max-num-seqs 32

The model's native context is 262,144 tokens; --max-model-len above is a memory-frugal default, not a limit of the checkpoint.

--max-num-seqs matters here: this is a hybrid Gated-DeltaNet model, which allocates one state cache block per decode sequence, and vLLM's default of 1024 overruns them and aborts CUDA-graph capture.

Speculative decoding (MTP)

The model's own NextN head drafts for it — no draft model to download:

vllm serve bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --max-model-len 49152 --max-num-seqs 32 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}'

Pin the drafter's attention backend. Left unset, vLLM picks FlashInfer for the MTP head, whose reorder-batch threshold (1) is taken as the minimum against the hybrid target's Triton backend (4); decode rows then queue behind continuing prefill chunks, the speculative state slots are never initialised, and long generations degenerate into repeated function words partway through — silently, without an error (vLLM #55894).

Sampling

Thinking mode, as the base model: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0.

Expected performance

Paired comparison with the bf16 source on the full quantization plan: both builds answer the same questions with the same seeds — RealWorldQA 765 questions × 2 seeds (images), GPQA-Diamond 198 × 4, MMLU-Pro 1,500 × 1 (a fixed slice), IFBench 300 × 2, AA-LCR 100 × 1 (89k–123k-token prompts, graded by Gemma-4-26B-A4B-it with thinking off). Thinking at the chat template's default reasoning effort (xhigh), sampled decoding (temperature 1.0, top_p 0.95, top_k 20, min_p 0.0), 65,536-token generation cap, vLLM 0.29.0. This build's answers were generated on an RTX PRO 6000 Blackwell, the bf16 reference on an H200.

Δ accuracy is this build minus bf16 on the same answers; its interval treats the question as the unit (seeds averaged per question first). Tokens are completion tokens (reasoning plus answer).

GPQA-Diamond seed 1 was generated with a 131,072-token cap and truncated to 65,536 afterwards; 2 of its 198 answers end at the cap.

Throughput — one RTX PRO 6000 Blackwell, vLLM 0.29.0, synthetic prompts of 1,024 tokens with 512 generated (last column: 32,768-token prompts, 128 generated), end-of-sequence ignored, prefix caching off, --max-num-seqs 64. Aggregate output tokens/s, median time to first token in ms in parentheses.

Against NVFP4, the weight-only 4-bit build, it is 25% faster at 64 concurrent requests, 74% faster with a 53% lower median time to first token on 32k-token prompts, and 29% slower for a single request. Pick it for batched serving on Blackwell; for one request at a time, or on a GPU without FP4 tensor cores, serve NVFP4.

Where to find us

<table style="border-collapse:collapse;border:0;margin:0"><tbody><tr> <td style="border:0;padding:0 18px 0 0"><a href="https://www.bottlecapai.com/"><img src="social-web.png" alt="Website" width="34" height="34"></a></td> <td style="border:0;padding:0 18px 0 0"><a href="https://www.linkedin.com/company/bottlecap-ai/"><img src="social-linkedin.png" alt="LinkedIn" width="34" height="34"></a></td> <td style="border:0;padding:0 18px 0 0"><a href="https://www.instagram.com/bottlecapai/"><img src="social-instagram.png" alt="Instagram" width="34" height="34"></a></td> <td style="border:0;padding:0"><a href="https://x.com/BottleCapAI"><img src="social-x.png" alt="X" width="34" height="34"></a></td> </tr></tbody></table>

Need even more efficiency? The open release is production-ready. Our enterprise versions go further — fewer thinking tokens still, tuned to your workload, at matched accuracy on your own tasks. Built for AI labs, inference providers and enterprises running models at scale. Deployed on your infrastructure, or in the cloud and region you choose. Talk to our team

License

ThinkingCap: PolyForm Small Business 1.0.0 + BottleCap personal-use grant (see LICENSE).

Upstream Qwen materials: Apache-2.0 (see NOTICE).

Commercial license: contact BottleCap AI.

Citation

If you use this model, please cite:

@misc{ThinkingCap-Qwen3.8-27B,
  title     = {bottlecapai/ThinkingCap-Qwen3.8-27B},
  author    = {Osusky, Adam and Lindauer, Jan and Jirkovsky, Adam and Mihal, Filip and Platek, Ondrej and Herel, David and Ihnatchenko, Luka and Bartek, Vojtech and Jirak, Jiri and Kubista, Daniel and Krus, Frantisek and Mikolov, Tomas},
  year      = {2026},
}