Downloads · 30 days
25
4% of all-time downloads
avtc/GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16
GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16 is a text generation model from avtc. Use it when you need the model to write or continue text. It is set up for vllm. The card lists the license as mit.
This is a 4-bit quantized (GPTQ) version of the GLM-4.6-REAP-268B-A32B model. It is able to run on 8x RTX 3090 (24GB) with -tp 8 and 200K tokens context window using fp8 KV cache.
Downloads · 30 days
25
4% of all-time downloads
All-time downloads
670
Public
Parameters
271B
145 GB on disk
Likes
3
Public
Click a slice to open those files.
.safetensors145 GB · 100%
How the weights are stored.
I32269B · 99%
From the Hugging Face model README
This is a 4-bit quantized (GPTQ) version of the GLM-4.6-REAP-268B-A32B model. It is able to run on 8x RTX 3090 (24GB) with -tp 8 and 200K tokens context window using fp8 KV cache.
c4/en (1024), arc (300), gsm8k (300), humaneval (164), and alpaca (20).export VLLM_ATTENTION_BACKEND="FLASHINFER"
export TORCH_CUDA_ARCH_LIST="8.6"
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export SAFETENSORS_FAST_GPU=1
vllm serve avtc/GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16 \
-tp 8 \
--port 8000 \
--host 0.0.0.0 \
--uvicorn-log-level info \
--trust-remote-code \
--gpu-memory-utilization 0.92 \
--max-num-seqs 1 \
--trust-remote-code \
--dtype=float16 \
--seed 1234 \
--max-model-len 202752 \
--tool-call-parser glm45 \
--reasoning-parser glm45 \
--enable-auto-tool-choice \
--enable-expert-parallel \
--enable-sleep-mode \
--compilation-config '{"level": 3, "cudagraph_capture_sizes": [1]}' \
--kv-cache-dtype fp8_e5m2
Recommended Sampling Parameters:
{
"top_p": 0.95,
"temperature": 0.8,
"repetition_penalty": 1.0,
"top_k": 40,
"min_p": 0.0
}
whole formatShows percent passed. Aider polyglot by default uses 2 tries.
| T | Pass 1 | Pass 2 | Pass 3 | Pass 4 | Pass 5 | Pass 6 | |
|---|---|---|---|---|---|---|---|
| Python | |||||||
| GLM-4.5-Air | |||||||
| avtc/GLM-4.5-Air-GPTQMODEL-W4A16 | 0.75 | 8.8 | 55.9 | 64.7 | 67.6 | 67.6 | 67.6 |
| avtc/GLM-4.5-Air-GPTQMODEL-W8A16 | 0.70 | 8.8 | 35.3 | 44.1 | 58.8 | 61.8 | 61.8 |
| z.ai CodingPlan GLM-4.5-Air | 0.80 | 5.9 | 38.2 | 55.9 | 67.6 | 67.6 | 70.6 |
| GLM-4.6 | |||||||
| avtc/GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16 | 0.80 | 23.5 | 61.8 | 73.5 | 94.1 | 94.1 | 94.1 |
| avtc/GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16-V2 | 0.00 | 20.6 | 73.5 | 91.2 | 100.0 | 100.0 | 100.0 |
| z.ai CodingPlan GLM-4.6 | 0.80 | 32.4 | 73.5 | 94.1 | 97.1 | 97.1 | 97.1 |
| z.ai CodingPlan GLM-4.6 | 1.00 | 23.5 | 76.5 | 85.3 | 85.3 | 91.2 | 94.1 |
| All languages | |||||||
| GLM-4.5-Air | |||||||
| avtc/GLM-4.5-Air-GPTQMODEL-W4A16 | 0.75 | 12.9 | 34.2 | 49.3 | 56.9 | 58.7 | 60.4 |
| GLM-4.6 | |||||||
| avtc/GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16-V2 | 0.00 | 19.1 | 58.7 | 80.4 | 85.8 | 87.1 | 88.0 |
| z.ai CodingPlan GLM-4.6 | 0.80 | 22.2 | 59.1 | 76.4 | 81.3 | 81.3 | 84.9 |
| z.ai CodingPlan GLM-4.6 | 1.00 | 23.1 | 61.8 | 74.2 | 80.0 | 85.8 | 88.0 |
Prompt:
Make an html animation of fishes in an aquarium. The aquarium is pretty, the fishes vary in colors and sizes and swim realistically. You can left click to place a piece of fish food in aquarium. Each fish chases a food piece closest to it, trying to eat it. Once there are no more food pieces, fishes resume swimming as usual.
Result: The model generated a working artifact using Kilo Code in Code mode with Temperature = 0.6 View the Result on JSFiddle
Special thanks to GPTQModel team for the quantization tools and support.
GLM-4.6-REAP-268B-A32B is a memory-efficient compressed variant of GLM-4.6 that maintains near-identical performance while being 25% lighter.
This model was created using REAP (Router-weighted Expert Activation Pruning), a novel expert pruning method that selectively removes redundant experts while preserving the router's independent control over remaining experts. Key features include:
GLM-4.6-REAP-268B-A32B has the following specifications:
Evaluations below refer to the original unquantized model.
For more details on the evaluation setup, refer to the REAP arXiv preprint.
The base checkpoint was created by applying the REAP (Router-weighted Expert Activation Pruning) method uniformly across all Mixture-of-Experts (MoE) blocks of GLM-4.6, with a 25% pruning rate.
REAP selects experts to prune based on a novel saliency criterion that considers both:
This dual consideration ensures that experts contributing minimally to the layer's output are pruned, while preserving those that play critical roles in the model's computations.
The model was calibrated using a diverse mixture of domain-specific datasets including:
📚 For more details, refer to the following resources:
This model is derived from
zai-org/GLM-4.6
and distributed under the MIT license.
If you use this checkpoint, please cite the REAP paper:
@article{lasby-reap,
title={REAP the Experts: Why Pruning Prevails for One-Shot MoE compression},
author={Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
journal={arXiv preprint arXiv:2510.13999},
year={2025}
}