Qwen3.5-99B-GGUF

[!TIP] Support this work → · X · GitHub · REAP paper · Cerebras REAP

Qwen3.5-99B-GGUF

GGUF quantization of 0xSero/Qwen3.5-99B.

At a glance

Which variant should I pick?

GGUF quantizations of 0xSero/Qwen3.5-99B, a 20% expert-pruned Qwen3.5-122B MoE model using REAP.

Available Quantizations

Model Details

Speed Benchmarks

Tested on AMD Ryzen AI MAX+ 395 (Strix Halo), Radeon 8060S (gfx1151), 128 GB LPDDR5X. llama.cpp b8746, Vulkan RADV, Flash Attention ON.

llama-bench (pp512 / tg128)

API Speed (llama-server, real chat completions)

Q6_K and Q8_0 are partially offloaded to CPU because they exceed the default 64 GB GTT limit. With GTT increased to 120 GB (BIOS GART + modprobe config), they would run at full GPU speed.

Quality Benchmarks

Tested via llama-server API with thinking mode enabled.

Reasoning (5 questions — math, calculus, logic, code comprehension, knowledge)

All quants produce correct answers for arithmetic (127*43=5461), calculus (derivative of x^3+2x^2-5x+7), formal logic, Python reference semantics, and factual recall.

Code Generation (HumanEval subset — 5 problems, executed and tested)

The model generates correct code for all problems. Score differences are due to code extraction from the thinking format, not model quality.

Full Benchmarks (safetensors, from base model card)

See the full model card for complete benchmark results and methodology.

How to Run

llama-server (recommended)

# Q4_K_M — fits in 64 GB, fastest
llama-server \
  -m Qwen3.5-122B-A10B-REAP-20-Q4_K_M.gguf \
  -ngl 999 --flash-attn on -c 4096 \
  --port 8080 --host 0.0.0.0

# With speculative decoding for faster generation
llama-server \
  -m Qwen3.5-122B-A10B-REAP-20-Q4_K_M.gguf \
  -ngl 999 --flash-attn on -c 4096 \
  --spec-type ngram-mod --spec-ngram-size-n 24 \
  --draft-min 48 --draft-max 64 \
  --port 8080 --host 0.0.0.0

Ollama

# Create a Modelfile
echo 'FROM ./Qwen3.5-122B-A10B-REAP-20-Q4_K_M.gguf' > Modelfile
ollama create reap20 -f Modelfile
ollama run reap20

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="Qwen3.5-122B-A10B-REAP-20-Q4_K_M.gguf",
    n_gpu_layers=-1,
    n_ctx=4096,
    flash_attn=True,
)

output = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=512,
)
print(output["choices"][0]["message"]["content"])

Which Quant Should I Use?

Hardware Notes

This model was designed for and tested on AMD Strix Halo (Ryzen AI MAX+ 395) with 128 GB unified memory. It also works on any system with sufficient VRAM/RAM:

• Strix Halo (64 GB GTT default): Q4_K_M fits fully, Q6_K/Q8_0 partial offload
• Strix Halo (120 GB GTT increased): All quants fit fully
• 2x Strix Halo cluster (RPC): All quants at full speed
• NVIDIA A100 80GB: Q4_K_M and Q6_K fit fully
• Apple M-series (128 GB): All quants should work via Metal

What is REAP?

REAP (Routing-Enhanced Activation Pruning) removes the least-activated experts from Mixture-of-Experts models while preserving critical capabilities. This model has 20% of experts removed (256 -> 205 per layer), retaining 97.9% average capability across standard benchmarks.

Credits

• Pruning: 0xSero / Sybil Solutions
• Base Model: Qwen Team
• REAP Method: arxiv:2510.13999
• Quantization: llama.cpp

License

Same license as the base model. See Qwen3.5-122B-A10B license.

License & citation

License inherited from the base model.

@misc{lasby2025reap,
  title  = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression},
  author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
  year   = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv}
}

Sponsors

Made possible by NVIDIA · TNG Technology · Lambda · Prime Intellect · Hot Aisle.