GLM-4.7-Flash-MXFP4

Note: If you have a multi-GPU SM120 Blackwell system (RTX 50/Pro), try my vLLM fork to resolve P2P / TP=2 issues (Pending PR into upstream).

Note: If you are running this MXFP4 model on SM120 GPU's, you also will need to use my fork until PR into upstream is merged, however it is significantly slower than NVFP4.

https://github.com/Gadflyii/vllm/tree/main

GLM-4.7-Flash MXFP4

This is a MXFP4 quantization of zai-org/GLM-4.7-Flash, a 30B-A3B (30B total, 3B active) Mixture-of-Experts model.

Quantization Strategy

This model uses MXFP4 (Microscaling FP4) format with the Marlin backend for inference. Custom quantization with calibration (128 samples, 2048 max seq len) applied to MoE experts.

MXFP4 vs NVFP4

Performance

Usage

Requirements

• vLLM: 0.14.0+ (for MXFP4 Marlin backend support)
• transformers: 5.0.0+ (for glm4_moe_lite architecture)
• GPU: NVIDIA GPU with compute capability 8.0+ (Ampere/Hopper/Blackwell)

Installation

pip install vllm>=0.14.0
pip install git+https://github.com/huggingface/transformers.git

Inference with vLLM

import os
os.environ["VLLM_MXFP4_USE_MARLIN"] = "1"

from vllm import LLM, SamplingParams

model = LLM(
    "GadflyII/GLM-4.7-Flash-MXFP4",
    tensor_parallel_size=1,
    max_model_len=65536,  # Can go up to 202K with sufficient VRAM
    trust_remote_code=True,
    gpu_memory_utilization=0.90,
)

# Note: Do NOT use repetition_penalty > 1.05, it causes degradation at long outputs
params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=2048)
outputs = model.generate(["Explain quantum computing in simple terms."], params)
print(outputs[0].outputs[0].text)

Serving with vLLM

VLLM_MXFP4_USE_MARLIN=1 vllm serve GadflyII/GLM-4.7-Flash-MXFP4 \
    --tensor-parallel-size 1 \
    --max-model-len 65536 \
    --trust-remote-code \
    --gpu-memory-utilization 0.90

Chat Completions API

import requests

payload = {
    "model": "GadflyII/GLM-4.7-Flash-MXFP4",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 1024,
    "temperature": 0.7,
    # Disable thinking mode for direct responses:
    "chat_template_kwargs": {"enable_thinking": False}
    # Or enable thinking for reasoning tasks:
    # "chat_template_kwargs": {"enable_thinking": True}
}
response = requests.post("http://localhost:8000/v1/chat/completions", json=payload)
print(response.json()["choices"][0]["message"]["content"])

Important Usage Notes

Sampling Parameters

Thinking Mode

This model supports a "thinking" mode where it shows its reasoning process:

• enable_thinking: True - Model outputs its reasoning process before the answer (good for math, coding, complex reasoning)
• enable_thinking: False - Model outputs the answer directly (good for chat, simple Q&A)

The model thinks in English when given English prompts.

Model Details

• Base Model: zai-org/GLM-4.7-Flash
• Architecture: Glm4MoeLiteForCausalLM
• Parameters: 30B total, 3B active per token (30B-A3B)
• MoE Configuration: 64 routed experts, 4 active, 1 shared expert
• Layers: 47
• Context Length: 202,752 tokens (max)
• Languages: English, Chinese

Quantization Details

• Format: MXFP4 (Microscaling FP4)
• Weight Format: E2M1 (4-bit floating point, range ±6.0)
• Scale Format: E8M0 (8-bit power-of-2 scales)
• Block Size: 32
• Calibration: 128 samples from neuralmagic/calibration dataset

Evaluation

MMLU-Pro Overall Results

MMLU-Pro by Category

MMLU-Pro by Subject (All 14 Subjects)

Citation

If you use this model, please cite the original GLM-4.7-Flash:

@misc{glm4flash2025,
  title={GLM-4.7-Flash},
  author={Zhipu AI},
  year={2025},
  howpublished={\url{https://huggingface.co/zai-org/GLM-4.7-Flash}}
}

License

This model inherits the Apache 2.0 license from the base model.