GLM-4.6-NVFP4

lukealonso/GLM-4.6-NVFP4

GLM-4.6-NVFP4 is a machine learning model from lukealonso. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.

Downloads · 30 days

358

6% of all-time downloads

All-time downloads

Public

Parameters

177B

201 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors201 GB · 100%

Parameter types

How the weights are stored.

U8176B · 88%

Base models

zai-org/GLM-4.6

Model card

From the Hugging Face model README

modelopt NVFP4 quantized GLM-4.6

Tested (but not extensively validated) on 4x RTX Pro 6000 Blackwell via:

 inference:
    image: vllm/vllm-openai:nightly
    container_name: inference
    ports:
      - "0.0.0.0:8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: -1          # use all GPUs
              capabilities: [gpu]
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
      - NCCL_IB_DISABLE=1
      - NCCL_NVLS_ENABLE=0
      - NCCL_P2P_DISABLE=0
      - NCCL_SHM_DISABLE=0
      - VLLM_USE_V1=1
      - VLLM_USE_FLASHINFER_MOE_FP4=1
      - OMP_NUM_THREADS=8
    volumes:
      - /models/GLM-4.6-NVFP4-4:/GLM-4.6:ro
    command:
      - /GLM-4.6
      - --enable-expert-parallel
      - --enable-prefix-caching
      - --enable-chunked-prefill
      - --served-model-name
      - "GLM-4.6"
      - --tensor-parallel-size
      - "4"
      - --gpu-memory-utilization
      - "0.95"
      - --max-num-batched-tokens
      - "16384"
      - --dtype
      - "auto"
      - --max-num-seqs
      - "8"
      - --kv-cache-dtype
      - fp8
      - --enable-auto-tool-choice
      - --tool-call-parser
      - glm45
      - --host
      - "0.0.0.0"
      - --port
      - "8000"```

Type: glm4_moe
License: mit
Created: Oct 24, 2025
Updated: Nov 3, 2025