GLM-4.7-Flash-MTP-NVFP4

Note: If you have a multi-GPU SM120 Blackwell system (RTX 50/Pro), try my vLLM fork to resolve P2P / TP=2 issues (Pending PR into upstream).

https://github.com/Gadflyii/vllm/tree/main

GLM-4.7-Flash-MTP-NVFP4 (Mixed Precision with MTP in BF16)

This is a mixed precision NVFP4 quantization of zai-org/GLM-4.7-Flash, a 30B-A3B (30B total, 3B active) Mixture-of-Experts model. This version preserves MTP (Multi-Token Prediction) layers in BF16 for speculative decoding compatibility.

What's Different from GLM-4.7-Flash-NVFP4?

Quantization Strategy

This model uses mixed precision to preserve accuracy and MTP functionality:

Performance

MTP Acceptance Rate

MTP quality is preserved (actually slightly improved) after quantization.

MTP Performance Note

MTP speculative decoding currently shows overhead rather than speedup due to missing torch.compile support for the MTP drafter model in vLLM. For best throughput, run without MTP enabled until this is resolved upstream.

Usage

Requirements

• vLLM: 0.8.0+ (for compressed-tensors NVFP4 support)
• transformers: 5.0.0+ (for glm4_moe_lite architecture)
• GPU: NVIDIA GPU with FP4 tensor core support (Blackwell, Hopper, Ada Lovelace)

Installation

pip install vllm>=0.8.0
pip install git+https://github.com/huggingface/transformers.git

Inference with vLLM (Recommended)

from vllm import LLM, SamplingParams

model = LLM(
    "GadflyII/GLM-4.7-Flash-MTP-NVFP4",
    tensor_parallel_size=1,
    max_model_len=4096,
    trust_remote_code=True,
    gpu_memory_utilization=0.90,
)

params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = model.generate(["Explain quantum computing in simple terms."], params)
print(outputs[0].outputs[0].text)

Serving with vLLM

# Standard serving (recommended for performance)
VLLM_ATTENTION_BACKEND=TRITON_MLA vllm serve GadflyII/GLM-4.7-Flash-MTP-NVFP4 \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --trust-remote-code \
    --gpu-memory-utilization 0.90

# With MTP speculative decoding (experimental)
VLLM_ATTENTION_BACKEND=TRITON_MLA vllm serve GadflyII/GLM-4.7-Flash-MTP-NVFP4 \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --trust-remote-code \
    --gpu-memory-utilization 0.90 \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Model Details

• Base Model: zai-org/GLM-4.7-Flash
• Architecture: Glm4MoeLiteForCausalLM
• Parameters: 30B total, 3B active per token (30B-A3B)
• MoE Configuration: 64 routed experts, 4 active, 1 shared expert
• Layers: 47 (with 1 MTP layer)
• Context Length: 202,752 tokens (max)
• Languages: English, Chinese

Quantization Details

• Format: compressed-tensors (NVFP4)
• Block Size: 16
• Scale Format: FP8 (E4M3)
• Calibration: 512 samples from wikitext dataset
• Calibration Sequence Length: 4096
• Full Expert Calibration: All 64 experts calibrated per sample

Tensors by Precision

Evaluation

MMLU-Pro Overall Results

MMLU-Pro by Category

MMLU-Pro by Subject

Citation

If you use this model, please cite the original GLM-4.7-Flash:

@misc{glm4flash2025,
  title={GLM-4.7-Flash},
  author={Zhipu AI},
  year={2025},
  howpublished={\url{https://huggingface.co/zai-org/GLM-4.7-Flash}}
}

License

This model inherits the Apache 2.0 license from the base model.