Downloads · 30 days
90
11% of all-time downloads
eisbaw/Qwen3.6-35B-A3B-NVFP4-gdn4lm4
Qwen3.6-35B-A3B-NVFP4-gdn4lm4 is a text generation model from eisbaw. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
RedHatAI/Qwen3.6-35B-A3B-NVFP4 with two more tensor groups quantised to NVFP4: the gated-delta-net (GDN/Mamba) backbone projections and the LM head. Weight-only (W4A16), via compressed-tensors config groups. Nothing e…
Downloads · 30 days
90
11% of all-time downloads
All-time downloads
797
Public
Parameters
34.7B
22.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors22.9 GB · 100%
How the weights are stored.
U834.1B · 98%
From the Hugging Face model README
RedHatAI/Qwen3.6-35B-A3B-NVFP4 with two more tensor groups quantised to NVFP4: the
gated-delta-net (GDN/Mamba) backbone projections and the LM head. Weight-only (W4A16), via
compressed-tensors config groups. Nothing else is changed.
Measured 2.26× single-stream decode on one NVIDIA DGX Spark (GB10) — 42.08 → 95.03 tok/s — with no measurable quality change on GSM8K.
The stock NVFP4 checkpoint quantises the MoE experts but leaves the always-active dense backbone
in BF16. On a bandwidth-bound box that is the wrong half to compress: an audit of per-decode-step
reads found ~80% of the bytes moved every step are the unquantised BF16 backbone
(mamba_in_proj alone is 1.52 GB) while the active NVFP4 experts are only ~0.56 GB. Decode at batch-1
is DRAM-bound, so tok/s tracks bytes-read-per-token almost exactly — and this checkpoint attacks the
term that actually dominates.
Single DGX Spark (GB10, sm_121a, 128 GB unified LPDDR5X, ~240 GB/s achievable read), vLLM 0.26.0, non-streaming, coherence-guarded.
| stock NVFP4 | this checkpoint | ||
|---|---|---|---|
| single-stream decode, no spec | 42.08 | 95.03 | 2.26× |
| aggregate @C=64 | 702.7 | 781.8 | 1.11× |
GSM8K, lm-eval n=200, 5-shot, apply_chat_template, greedy:
| strict-match | flexible-extract | |
|---|---|---|
| stock NVFP4 | 0.945 | 0.955 |
| this checkpoint | 0.950 | 0.965 |
No measurable degradation. (The small deltas are within noise at n=200; the honest claim is "no measurable change", not "better".)
vllm serve <this-repo> \
--kv-cache-dtype fp8 \
--reasoning-parser qwen3 --enable-prefix-caching
Two flags worth knowing on this model family, both independent of this checkpoint:
--mamba-ssm-cache-dtype float16 — the config declares mamba_ssm_dtype: float32, and the GDN
recurrent state is the only per-step term that never amortises across concurrent rows. At high
concurrency it dominates: halving it measured +51% aggregate at C=512. Note the state is a
recurrent accumulator, so this is gated on short-generation GSM8K only — long-context behaviour
is unverified.--max-num-seqs — vLLM's default is 128; raising it is worth a lot on this model if you are
serving many streams.compressed-tensors config groups on the existing checkpoint — no vLLM patch, no retraining:
group_0 — stock W4A4 (unchanged, targets: ["Linear"])group_1 — LM head, weight-only: re:.*lm_head$group_2 — GDN projections, weight-only:
re:.*linear_attn\.in_proj_qkv$, re:.*linear_attn\.in_proj_z$,
re:.*linear_attn\.out_proj$, re:.*linear_attn\.in_proj_qkvz$re:.*linear_attn\.in_proj_qkvz$ is mandatory. Scheme selection runs against the runtime
fused module name. Listing only the per-shard names (in_proj_qkv, in_proj_z) matches nothing,
and the module silently falls back to group_0's W4A4 — which has no calibrated
input_global_scale, so the model loads and serves fluent garbage. If you fork this recipe,
check for the absence of the global-scale warning in the serve log; that absence is the signal.in_proj_ba to group_2. Its shards stay BF16, and a quantised scheme would
demand packed tensors that do not exist.Same reason group_1 uses a regex: targets: ["Linear"] matches by module class, and
ParallelLMHead is not a LinearBase, so the head is not covered by group_0.
All model work is upstream: Qwen for Qwen3.6-35B-A3B and RedHatAI for the NVFP4 conversion this builds on. This repository contributes only the two additional tensor groups and the measurements above. Apache-2.0, inherited from the base model.