GLM-4.7-Flash-REAP-23B-A3B-NVFP4

GLM-4.7-Flash-REAP-23B-A3B-NVFP4

NVFP4 PTQ checkpoint of cerebras/GLM-4.7-Flash-REAP-23B-A3B, produced with NVIDIA TensorRT Model Optimizer (hf_ptq).

This is a community quantized derivative. Credit for the base weights belongs to Cerebras (REAP prune) and Z.ai (original GLM-4.7-Flash). Please cite / attribute the upstream models when using this checkpoint.

Lineage

zai-org/GLM-4.7-Flash
        │
        │  Cerebras REAP — uniform 25% expert prune
        │  (arxiv:2510.13999)
        ▼
cerebras/GLM-4.7-Flash-REAP-23B-A3B   (~23B-A3B MoE, BF16)
        │
        │  ModelOpt PTQ — NVFP4 on routed experts
        │  (attention / gates / shared experts / embeds kept high-prec)
        ▼
dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4   (this repo)

Quantization recipe (what changed)

From hf_quant_config.json (quant_algo: NVFP4, group_size: 16):

Note: ModelOpt calibration initially enabled FP8 KV scales; the published export has KV cache quantization disabled (GB10 / Spark-friendly: serve with --kv-cache-dtype auto).

Intended use

Same as the REAP MoE base, with much lower memory footprint on Blackwell GPUs (e.g. DGX Spark GB10). NVFP4 weight kernels need Blackwell.

Serve with vLLM

Validated on DGX Spark with vllm/vllm-openai:nightly-aarch64:

vllm serve dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4 \
  --quantization modelopt_fp4 \
  --moe-backend marlin \
  --enforce-eager \
  --kv-cache-dtype auto \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.161 \
  --served-model-name GLM-4.7-Flash-REAP-23B-A3B-NVFP4

Notes:

• --moe-backend marlin was required for this MoE NVFP4 checkpoint on the tested nightly.
• --enforce-eager and --kv-cache-dtype auto match the GB10 smoke/bench config (FP8 KV disabled).

DGX Spark speedups (BF16 → NVFP4)

vLLM on GB10, max-model-len=4096, concurrency 1.

Files to ignore in this local folder

If uploading from a local checkout that still has working artifacts, do not upload:

• config.json.pre_kv_patch
• hf_quant_config.json.pre_kv_patch
• .quant_summary.txt

Disclaimer

This checkpoint is provided as-is for experimentation. Quantization can change model quality; run your own evals before production use. Not affiliated with Cerebras, Z.ai, or NVIDIA beyond use of open ModelOpt tooling.