Downloads · 30 days
1.1K
2% of all-time downloads
scottgl/Qwen3.5-122B-A10B-NVFP4-GB10
Qwen3.5-122B-A10B-NVFP4-GB10 is a text generation model from scottgl. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Qwen3.5-122B-A10B optimized for NVIDIA DGX Spark (GB10, SM12.1) — all quantizations baked in.
Downloads · 30 days
1.1K
2% of all-time downloads
All-time downloads
47.8K
Public
Parameters
26.8B
79.7 GB on disk
Likes
7
Public
Click a slice to open those files.
.safetensors79.7 GB · 100%
How the weights are stored.
I3214.7B · 55%
From the Hugging Face model README
Qwen3.5-122B-A10B optimized for NVIDIA DGX Spark (GB10, SM12.1) — all quantizations baked in.
This checkpoint is derived from Sehyo/Qwen3.5-122B-A10B-NVFP4 with additional post-quantizations applied and saved permanently into the weights. The result loads and serves immediately on GB10 without any runtime quantization overhead.
The base NVFP4 checkpoint leaves several layer types in BF16 to keep it hardware-agnostic. On GB10 (NVIDIA Grace Blackwell, SM12.1), those BF16 layers are a significant bottleneck:
These optimizations are implemented in scottgl9/sglang-spark-gb10-optimizations as runtime post-quantization hooks. This checkpoint bakes them in so:
| Layer | Original | Baked format | Why |
|---|---|---|---|
| MoE expert weights (gate/up/down) | NVFP4 (W4A4) | NVFP4 (W4A4) | From source checkpoint — preserved as-is |
GDN in_proj_qkv, in_proj_z | BF16 | NVFP4 (Marlin FP4) | Large projections; SM12.1 has native Marlin FP4 kernels. CUTLASS FP4 is broken on GB10. |
GDN in_proj_a, in_proj_b, out_proj | BF16 | FP8 (per-tensor, dynamic activations) | Smaller GDN projections; FP8 CUTLASS works correctly on SM12.1. cos_sim > 0.999, SNR = 31.5 dB vs BF16. |
| lm_head (vocabulary projection) | BF16 | FP8 (dynamic activations) | 3072 x 248,320 matmul at every decode step. FP8 halves DRAM reads vs BF16. Activations remain dynamic per-token for accuracy. |
| Attention Q/K/V/O | BF16 | BF16 | Standard attention; not the bandwidth bottleneck at batch=1. |
~46 tok/s decode on a single NVIDIA DGX Spark (GB10, 128 GB unified memory) using SGLang with NEXTN speculative decoding (steps=2, draft_tokens=2).
Baseline without any of these optimizations: ~28 tok/s. The baked quantizations account for approximately +18 tok/s of that gain.
Tested on: NVIDIA DGX Spark (ASUS Ascent GX10), GB10 Grace Blackwell, SM12.1, 128 GB unified memory.
Requirements:
Will NOT work on:
# Load directly — no post-quant overhead at startup
SGLANG_QUANTIZE_LM_HEAD_FP8=0 ./sglang.sh Qwen3.5-NVFP4 \
--model-path scottgl9/Qwen3.5-122B-A10B-NVFP4-GB10
SGLANG_QUANTIZE_LM_HEAD_FP8=0 tells SGLang to skip the lm_head FP8 post-quant (already baked in). All other sglang.sh flags remain unchanged.
With NEXTN speculative decoding (recommended):
SGLANG_QUANTIZE_LM_HEAD_FP8=0 ./sglang.sh Qwen3.5-NVFP4 \
--model-path scottgl9/Qwen3.5-122B-A10B-NVFP4-GB10 \
--speculative-algorithm NEXTN \
--speculative-num-steps 2 \
--speculative-num-draft-tokens 2
Apache 2.0 (same as Qwen3.5)