Downloads · 30 days
20
3% of all-time downloads
avtc/GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16-V2
GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16-V2 is a text generation model from avtc. Use it when you need the model to write or continue text. It is set up for vllm. The card lists the license as mit.
This is a 4-bit quantized (GPTQ) version of the GLM-4.6-REAP-268B-A32B model. It is able to run on 8x RTX 3090 (24GB) with -tp 8 and 200K tokens context window using fp8 KV cache.
Downloads · 30 days
20
3% of all-time downloads
All-time downloads
581
Public
Parameters
271B
145 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors145 GB · 100%
How the weights are stored.
I32269B · 99%
From the Hugging Face model README
This is a 4-bit quantized (GPTQ) version of the GLM-4.6-REAP-268B-A32B model. It is able to run on 8x RTX 3090 (24GB) with -tp 8 and 200K tokens context window using fp8 KV cache.
c4/en (1024), ai2_arc (256), gsm8k (256), humaneval-x (320), alpaca (32), evol-codealpaca-v1 (32), SWE-smith-trajectories (1), WebSight (64), c-sharp-coding-dataset (64), LCC_csharp (4) in total 665K tokens.export VLLM_ATTENTION_BACKEND="FLASHINFER"
export TORCH_CUDA_ARCH_LIST="8.6"
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export SAFETENSORS_FAST_GPU=1
vllm serve avtc/GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16 \
-tp 8 \
--port 8000 \
--host 0.0.0.0 \
--uvicorn-log-level info \
--trust-remote-code \
--gpu-memory-utilization 0.92 \
--max-num-seqs 1 \
--trust-remote-code \
--dtype=float16 \
--seed 1234 \
--max-model-len 202752 \
--tool-call-parser glm45 \
--reasoning-parser glm45 \
--enable-auto-tool-choice \
--enable-expert-parallel \
--enable-sleep-mode \
--compilation-config '{"level": 3, "cudagraph_capture_sizes": [1]}' \
--kv-cache-dtype fp8_e5m2
Recommended Sampling Parameters:
It looks like it works fine with temperature from 0.0 to 1.0, sometimes require to validate or fix task. The Aider Polyglot works best with Temperature 0.0 (tested against 0.8 and 1.0), it is on pair with full model available on Coding Plan. Subjectively it seems with Temperature 0.0 it is even more consistent and function rich, but miss some work to be finished from the first attempt (so require more attempts to finish), while with Temperature 0.8-0.9 it tries to do less but done nice and properly.
{
"top_p": 0.95,
"temperature": 0.8,
"repetition_penalty": 1.05,
"top_k": 40,
"min_p": 0.05
}
whole formatShows percent passed. Aider polyglot by default uses 2 tries.
| T | Pass 1 | Pass 2 | Pass 3 | Pass 4 | Pass 5 | Pass 6 | |
|---|---|---|---|---|---|---|---|
| Python | |||||||
| GLM-4.5-Air | |||||||
| avtc/GLM-4.5-Air-GPTQMODEL-W4A16 | 0.75 | 8.8 | 55.9 | 64.7 | 67.6 | 67.6 | 67.6 |
| avtc/GLM-4.5-Air-GPTQMODEL-W8A16 | 0.70 | 8.8 | 35.3 | 44.1 | 58.8 | 61.8 | 61.8 |
| z.ai CodingPlan GLM-4.5-Air | 0.80 | 5.9 | 38.2 | 55.9 | 67.6 | 67.6 | 70.6 |
| GLM-4.6 | |||||||
| avtc/GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16 | 0.80 | 23.5 | 61.8 | 73.5 | 94.1 | 94.1 | 94.1 |
| avtc/GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16-V2 | 0.00 | 20.6 | 73.5 | 91.2 | 100.0 | 100.0 | 100.0 |
| z.ai CodingPlan GLM-4.6 | 0.80 | 32.4 | 73.5 | 94.1 | 97.1 | 97.1 | 97.1 |
| z.ai CodingPlan GLM-4.6 | 1.00 | 23.5 | 76.5 | 85.3 | 85.3 | 91.2 | 94.1 |
| All languages | |||||||
| GLM-4.5-Air | |||||||
| avtc/GLM-4.5-Air-GPTQMODEL-W4A16 | 0.75 | 12.9 | 34.2 | 49.3 | 56.9 | 58.7 | 60.4 |
| GLM-4.6 | |||||||
| avtc/GLM-4.6-REAP-268B-A32B-GPTQMODEL-W4A16-V2 | 0.00 | 19.1 | 58.7 | 80.4 | 85.8 | 87.1 | 88.0 |
| z.ai CodingPlan GLM-4.6 | 0.80 | 22.2 | 59.1 | 76.4 | 81.3 | 81.3 | 84.9 |
| z.ai CodingPlan GLM-4.6 | 1.00 | 23.1 | 61.8 | 74.2 | 80.0 | 85.8 | 88.0 |
Prompt:
Make an html animation of fishes in an aquarium. The aquarium is pretty, the fishes vary in colors and sizes and swim realistically. You can left click to place a piece of fish food in aquarium. Each fish chases a food piece closest to it, trying to eat it. Once there are no more food pieces, fishes resume swimming as usual.
Result: The model generated a working artifact using Kilo Code in Code mode with Temperature = 0.0
Initial result: View the Result on JSFiddle
After two additional requests in the same task:
2) why some food pieces disappear when other eaten? 3) add flora, sand, rocks, nice glowing effects
Special thanks to GPTQModel team for the quantization tools and support.
GLM-4.6-REAP-268B-A32B is a memory-efficient compressed variant of GLM-4.6 that maintains near-identical performance while being 25% lighter.
This model was created using REAP (Router-weighted Expert Activation Pruning), a novel expert pruning method that selectively removes redundant experts while preserving the router's independent control over remaining experts. Key features include:
GLM-4.6-REAP-268B-A32B has the following specifications:
Evaluations below refer to the original unquantized model.
For more details on the evaluation setup, refer to the REAP arXiv preprint.
The base checkpoint was created by applying the REAP (Router-weighted Expert Activation Pruning) method uniformly across all Mixture-of-Experts (MoE) blocks of GLM-4.6, with a 25% pruning rate.
REAP selects experts to prune based on a novel saliency criterion that considers both:
This dual consideration ensures that experts contributing minimally to the layer's output are pruned, while preserving those that play critical roles in the model's computations.
The model was calibrated using a diverse mixture of domain-specific datasets including:
📚 For more details, refer to the following resources:
This model is derived from
zai-org/GLM-4.6
and distributed under the MIT license.
If you use this checkpoint, please cite the REAP paper:
@article{lasby-reap,
title={REAP the Experts: Why Pruning Prevails for One-Shot MoE compression},
author={Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
journal={arXiv preprint arXiv:2510.13999},
year={2025}
}