GLM-4.7-Flash-REAP-23B-A3B-NVFP4-NVembed

GLM-4.7-Flash-REAP-23B-A3B-NVFP4-NVembed

NVFP4-quantized GLM-4.7-Flash-REAP-23B-A3B — a 23B-parameter MoE reasoning model (3B active) that fits in 12 GB VRAM and runs 44K context on a single RTX 5080.

What's special

• Full NVFP4: all Linear layers, lm_head, and embed_tokens quantized to NVFP4
• FP8 KV cache: MLA attention with 512-dim compressed KV, stored in FP8
• 44,000-token context on 16 GB VRAM (vs ~11K without lm_head/embed quantization)
• 12.1 GiB single-file model.safetensors
• Thinking/reasoning support (<think> tokens) with --reasoning-parser deepseek_r1

Hardware

• GPU: NVIDIA RTX 5080 (16 GB VRAM, SM 12.0 / Blackwell)
• NVFP4 GEMM: Marlin backend (FlashInfer FP4 JIT fails on SM 12.0)
• MLA backend: Triton MLA (FLASHINFER_MLA requires qk_nope_head_dim=128, this model has 192)

Should work on any Blackwell GPU (RTX 50-series, B-series datacenter). Context length scales with available VRAM.

Quick start

1. Clone and build vLLM

git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout 628302114  # v0.16.1rc1.dev34

# Apply patches (required for NVFP4 lm_head, embed_tokens, MLA FP8 KV cache)
git apply ../GLM-4.7-Flash-REAP-23B-A3B-NVFP4-NVembed/vllm_patches.diff

# Build
pip install -e . --no-build-isolation

2. Install dependencies

pip install flashinfer==0.6.4  # or latest compatible

3. Serve

export VLLM_TEST_FORCE_FP8_MARLIN=1
export VLLM_NVFP4_GEMM_BACKEND=marlin
export PYTORCH_ALLOC_CONF=expandable_segments:True
export VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE=$((64 * 1024 * 1024))

BLOCKS=2750
BYTES_PER_BLOCK=433152  # 47 layers x 576 MLA head x 1 FP8 x 16 tokens
MAX_MODEL_LEN=$((BLOCKS * 16))  # 44,000

vllm serve ./GLM-4.7-Flash-REAP-23B-A3B-NVFP4-NVembed \
    --trust-remote-code \
    --dtype bfloat16 \
    --quantization modelopt \
    --kv-cache-dtype fp8_e4m3 \
    --max-model-len $MAX_MODEL_LEN \
    --no-enable-prefix-caching \
    --max-num-seqs 1 \
    --kv-cache-memory-bytes $((BLOCKS * BYTES_PER_BLOCK)) \
    --num-gpu-blocks-override $BLOCKS \
    --tensor-parallel-size 1 \
    --enable-chunked-prefill \
    --max-num-batched-tokens 256 \
    --enforce-eager \
    --reasoning-parser deepseek_r1 \
    --override-generation-config '{"temperature": 0.0, "max_tokens": 44000}'

4. Test

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "GLM-4.7-Flash-REAP-23B-A3B-NVFP4-NVembed",
       "messages": [{"role": "user", "content": "What is the capital of France?"}]}' \
  | python3 -m json.tool

Serve command flags explained

Performance

Quantization details

• Method: NVIDIA Model Optimizer (nvidia-modelopt==0.41.0)
• Format: NVFP4 (4-bit floating point, group_size=16)
• Calibration: 512 samples across 4 task types (code, instruction, agentic, structured data from databricks-dolly-15k)
• lm_head + embed_tokens: also NVFP4 (saves ~928 MB combined)
• Base model: THUDM/GLM-4.7-Flash-REAP-23B-A3B

vLLM patches

The included vllm_patches.diff modifies 6 files based on vLLM commit 628302114:

Upstream PRs

Some of these patches have been submitted upstream and may become unnecessary in future vLLM releases:

• #35576 — MLA weight access crash fix for quantized layers
• #35660 — NVFP4-quantized lm_head support