GLM-5.3-Nota-NVFP4-Global-Pruned-17.75

Nota Global-Pruned GLM-5.3 (17.75%) · NVFP4

Nota AI presents a 4-bit quantized and expert-pruned release of Z.ai's GLM-5.3 — a 753B-parameter Mixture-of-Experts model with ~42B active per token — produced with Nota AI's proprietary quantization method specialized for Mixture-of-Experts (MoE) large language models.

On top of NVFP4 quantization, this variant applies additional expert pruning so that the model fits on fewer GPUs. Rather than removing the same number of experts uniformly from every block, Nota AI's proprietary global importance-estimation method measures expert importance across the whole network and keeps only the most important experts on a per-layer basis (global pruning). As a result, the number of routed experts varies from layer to layer, preserving accuracy far better than uniform pruning while shrinking the memory footprint enough to serve on 2×B300 GPUs — down from 8 for the BF16 base model.

<div align="center">

8 × B300 → 2 × B300

Full 1,048,576-token context on both

</div>

Method clarification. This model does not use uniform expert pruning, in which the same number or proportion of experts is removed from every layer. Instead, we use our proprietary global-scale expert importance score to compare experts across the entire network and determine a different number of retained experts for each layer. Consequently, the pruned architecture has layer-wise variable expert counts, which are explicitly recorded in the model configuration and require the patched vLLM model definition provided in this repository.

Highlights

• NVFP4 (4-bit float, W4A4) — group_size=16, packed in the compressed-tensors format for direct serving in vLLM. Both weights and activations are quantized to 4-bit floating point.

  • Requires NVIDIA Blackwell. NVFP4 relies on the FP4 tensor cores introduced in the Blackwell architecture (e.g. B200 / B300 / GB200), so inference must run on a Blackwell-class GPU. Earlier architectures (Hopper, Ada, Ampere) do not support NVFP4 execution.

  • Only the routed experts are quantized. MLA attention, the DSA indexer, routers, shared experts, the leading dense MLPs, norms, embeddings and the MTP block stay in BF16 — the same split the reference NVFP4 release uses.
• Global expert pruning. Nota AI's proprietary global importance-estimation method prunes experts non-uniformly across layers — keeping only the experts that matter most in each layer — so the quantized model runs on fewer GPUs with minimal accuracy loss. Across the 75 MoE layers each block keeps between 208 and 256 experts, 15,792 in total out of the original 19,200 (17.75% pruned). The per-layer counts ship in config.json as num_experts_per_layer.
• Nota AI's proprietary MoE quantization method. The release is produced with Nota AI's quantization stack for Mixture-of-Experts models, developed to preserve model quality under aggressive low-bit quantization — including MoE-aware calibration and routing-preserving quantization of the routed experts.

Performance

• Both gaps sit inside the sampling noise of a single run. With 89 and 113 tasks at k=1 the standard error is 4.0 and 4.5 points, and the 95% intervals — 75.4–90.9 and 54.9–72.6 — both contain the reference score, so neither difference is statistically distinguishable from no change. Relative retention is 94.3% and 95.2% at 26% of the BF16 footprint.
• The run-to-run spread is that large in practice: re-running eleven already-passing tasks under identical settings flipped three of them to failure, which is expected at temperature=1.0.

Evaluation setup

Identical to the original model card's footnotes.

Installation

Built against the vLLM version from the official GLM-5.3 recipe, installed as documented there (as of 2026-08-31):

vllm 0.28.0

Then put patch/deepseek_v2.py from this repository in place of vLLM's own deepseek_v2.py:

hf download nota-ai/GLM-5.3-Nota-NVFP4-Global-Pruned-17.75 patch/deepseek_v2.py --local-dir .

TARGET="$(python3 -c 'import vllm, pathlib; print(pathlib.Path(vllm.__file__).parent / "model_executor/models/deepseek_v2.py")')"
cp "${TARGET}" "${TARGET}.orig"     # keep the stock file for restoring
cp patch/deepseek_v2.py "${TARGET}"

vLLM serves GLM-5.3 through its DeepSeek modeling file — GlmMoeDsaForCausalLM is registered to deepseek_v2 and defined there as a subclass of DeepseekV2ForCausalLM — so that is the file to replace, not a GLM one.

The patch swaps config.n_routed_experts for that layer's count only while a decoder block is being constructed — DeepseekV2MoE.__init__ reads the scalar for the gate width, the e_score_correction_bias size and the FusedMoE expert count, and never reads it again. A checkpoint without num_experts_per_layer passes through untouched, so the patched file also serves stock DeepSeek and GLM models. Only model.layers.* are overridden; the MTP head is not pruned and keeps reading the scalar.

On a different vLLM version, do not copy the file over. Port the _nu_per_layer_num_experts block and the one with line in DeepseekV2DecoderLayer.__init__ into that version's file.

Quick Start

B300 × 2

VLLM_NO_USAGE_STATS=1 CUDA_VISIBLE_DEVICES=0,1 \
vllm serve nota-ai/GLM-5.3-Nota-NVFP4-Global-Pruned-17.75 \
  --served-model-name GLM-5.3-Nota-NVFP4-Global-Pruned-17.75 \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.96 \
  --kv-cache-dtype fp8_e4m3 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice --tool-call-parser glm47

--gpu-memory-utilization 0.96 matters: vLLM's CUDA-graph memory profiling makes the default 0.90 behave like 0.879, and at that budget the KV cache holds 345,920 tokens — not enough for a single full-length request. At 0.96 the cache holds 1,590,400 tokens, so the model serves its full 1,048,576-token context without --max-model-len.

--enable-expert-parallel is safe here: EP requires each layer's expert count to divide by the EP size, and every layer in this checkpoint is a multiple of 16.

Do not add --enable-eplb. vLLM reads the expert count of the first MoE block and assumes every layer matches, which a non-uniform checkpoint breaks.

GLM-5.3's reasoning_effort parameter (low / high / max) works unchanged.

Patch Files for vLLM