Downloads · 30 days
191
15% of all-time downloads
nota-ai/Qwen3.8-2.4T-A95B-Nota-NVFP4-Global-Pruned-40
Qwen3.8-2.4T-A95B-Nota-NVFP4-Global-Pruned-40 is a text generation model from nota-ai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
Nota AI presents a 4-bit quantized and expert-pruned release of Qwen's Qwen3.8-2.4T-A95B, produced with Nota AI's proprietary quantization technology specialized for Mixture-of-Experts (MoE) large language models.
Downloads · 30 days
191
15% of all-time downloads
All-time downloads
1.2K
Public
Parameters
1.5T
1 TB on disk
Likes
24
Public
Click a slice to open those files.
.safetensors1 TB · 100%
How the weights are stored.
U81.4T · 95%
From the Hugging Face model README
Nota AI presents a 4-bit quantized and expert-pruned release of Qwen's Qwen3.8-2.4T-A95B, produced with Nota AI's proprietary quantization technology specialized for Mixture-of-Experts (MoE) large language models.
On top of NVFP4 quantization, this variant applies additional expert pruning so that the model fits on fewer GPUs. Rather than removing the same number of experts uniformly from every block, Nota AI's proprietary global importance-estimation technique measures expert importance across the whole network and keeps only the most important experts on a per-layer basis (global pruning). As a result, the number of routed experts varies from layer to layer, preserving accuracy far better than uniform pruning while shrinking the memory footprint enough to serve on 4×B300 GPUs — down from 24 for the BF16 base model.
Method clarification. This model does not use REAP-style uniform expert pruning, in which the same number or proportion of experts is removed from every layer. Instead, we use our proprietary global-scale expert importance score to compare experts across the entire network and determine a different number of retained experts for each layer. Consequently, the pruned architecture has layer-wise variable expert counts, which are explicitly recorded in the model configuration and require the patched vLLM model definition provided in this repository.
group_size=16, packed in the llm-compressor
(compressed-tensors) format for direct serving in
vLLM. Both weights and activations are quantized to
4-bit floating point.
Requires NVIDIA Blackwell. NVFP4 relies on the FP4 tensor cores introduced in the Blackwell architecture (e.g. B200 / B300 / GB200), so inference must run on a Blackwell-class GPU. Earlier architectures (Hopper, Ada, Ampere) do not support NVFP4 execution.
config.json as
num_experts_per_layer.| Model | AIME | GPQA-Diamond | HumanEval | HumanEval+ | IFEval |
|---|---|---|---|---|---|
| NVFP4 (no pruning) | 90.0 | 83.8 | 98.8 | 94.5 | 91.3 |
| Nota Global-Pruned Qwen3.8-2.4T-A95B (40%) · NVFP4 | 90.0 | 84.9 | 98.2 | 93.3 | 92.1 |
Metrics used: exact_match for AIME (mean of AIME 2024 and AIME 2025),
exact_match (flexible-extract) for GPQA-Diamond, pass@1 for HumanEval and HumanEval+,
prompt_level_strict_acc for IFEval.
temperature=0, top_p=1.0, top_k=-1, seed=0),
reasoning_effort=medium and a 65K generation budget, so that the run is reproducible and the
gap is attributable to pruning alone. Under the base model's recommended sampling and a larger
budget the scores are higher.Built against the vLLM version from the official Qwen3.8-2.4T-A95B recipe, installed as documented there (as of 2026-08-18):
vllm 0.1.dev19754+g3a0914114
Then put patch/qwen3_5.py from this repository in place of vLLM's own qwen3_5.py:
hf download nota-ai/Qwen3.8-2.4T-A95B-Nota-NVFP4-Global-Pruned-40 patch/qwen3_5.py --local-dir .
TARGET="$(python3 -c 'import vllm, pathlib; print(pathlib.Path(vllm.__file__).parent / "model_executor/models/qwen3_5.py")')"
cp "${TARGET}" "${TARGET}.orig" # keep the stock file for restoring
cp patch/qwen3_5.py "${TARGET}"
The patch swaps config.num_experts for that layer's count only while a decoder block is being
constructed — Qwen3NextSparseMoeBlock.__init__ reads the scalar for the gate width, the
FusedMoE expert count and the TP check, and never reads it again. A checkpoint without
num_experts_per_layer passes through untouched, so the patched file also serves stock Qwen3.5
models. Only model.layers.* are overridden; the MTP head is not pruned and keeps reading the
scalar.
On a different vLLM version, do not copy the file over. Port the _nu_per_layer_num_experts
block and the one with line in Qwen3_5DecoderLayer.__init__ into that version's file.
B300 × 4
CUDA_VISIBLE_DEVICES=0,1,2,3 \
vllm serve nota-ai/Qwen3.8-2.4T-A95B-Nota-NVFP4-Global-Pruned-40 \
--served-model-name Qwen3.8-2.4T-A95B-Nota-NVFP4-Global-Pruned-40 \
--trust-remote-code \
--load-format fastsafetensors --safetensors-load-strategy lazy \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--max-model-len 262144 \
--max-num-seqs 32 \
--gpu-memory-utilization 0.90 \
--kv-cache-dtype fp8 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder
--max-num-seqs 32 matters: 69 of the 92 layers are gated-delta-net (linear attention) and their
recurrent state is pre-allocated per sequence out of the same pool as the attention KV cache —
about 138 MiB per sequence on this 4-GPU layout. vLLM's default of 1024 exhausts the pool before
a single token is cached and aborts during CUDA-graph capture.
--enable-expert-parallel is safe here: EP requires each layer's expert count to divide by the EP
size, and every layer in this checkpoint is a multiple of 8.
Do not add --enable-eplb. vLLM reads the expert count of the first MoE block and assumes every
layer matches, which a non-uniform checkpoint breaks.
| Path | Role |
|---|---|
patch/qwen3_5.py | vLLM modeling file that reads the per-layer expert counts |
config.json | num_experts_per_layer holds the 92 per-layer counts |
kept_experts.json | which original expert indices survived, per layer |