GLM-4.7-REAP-40p-GGUF

GLM-4.7-REAP-40p GGUF

Quantized GGUF versions of 0xSero/GLM-4.7-REAP-40, a REAP-pruned parameter Mixture-of-Experts model.

Model Details

• Base Model: zai-org/GLM-4.7 (355B total)
• Pruning: REAP (Router-Experts Activation Pruning) - 96 experts with 40% active
• Architecture: Mixture-of-Experts (MoE) with QK-RMS normalization
• Quantization: GGUF format for llama.cpp inference

Usage with llama.cpp

# Install llama.cpp (requires b7618 or later)
# Download from https://github.com/ggml-org/llama.cpp/releases

# Run inference
llama-cli -m GLM-4.7-REAP-40p-Q5_K_M.gguf \
  -p "Your prompt here" \
  -n 512 \
  -c 4096 \
  -t 8

Python API

from llama_cpp import Llama

llm = Llama(
    model_path="GLM-4.7-REAP-40p-Q5_K_M.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,  # offload all to GPU
    verbose=True
)

output = llm(
    "Q: What is the capital of France? A:",
    max_tokens=128,
    stop=["Q:", "\n"],
    echo=True
)
print(output['choices'][0]['text'])

Important Notes

NextN Layer Fix

The original REAP model config has num_nextn_predict_layers=1 but the actual weights don't contain a NextN layer (it was pruned away). These GGUF files have been converted with the corrected config (num_nextn_predict_layers=0), so they will load without errors.

Performance

• CPU-only: ~0.8 tokens/sec (Q3_K_S on modern CPU)
• GPU offload: Recommended for usable inference speeds
• VRAM requirements: Model size + ~2GB overhead for context

Original Model

For the original FP16 HuggingFace format, see:

• 0xSero/GLM-4.7-REAP-40

License

This model inherits the license from the base model. See GLM-4 License for details.

Quantization Details

All quantizations use --output-tensor-type Q8_0 for the output layer to preserve generation quality. Quantization was performed using llama.cpp b7618.

• Conversion: HuggingFace → GGUF FP16 → Quantized GGUF
• Tool: llama.cpp b7618
• Platform: Google Cloud TPU v6e (spot instance)