Qwen3.5-264B

[!TIP] Support this work → · X · GitHub · REAP paper · Cerebras REAP

Qwen3.5-264B

REAP-pruned Qwen/Qwen3.5-397B-A17B.

At a glance

Which variant should I pick?

• Repository: 0xSero/Qwen3.5-264B
• Base model: Qwen/Qwen3.5-397B-A17B
• Artifact kind: pruned
• Compression ratio: 34%
• Prune metric: reap

Details

• Maintainer: 0xSero
• Organization: Sybil Solutions
• Project: REAP PR17
• Hub owner: 0xSero
• Summary: BF16 REAP-pruned Qwen3.5-397B-A17B with 176 of 512 experts removed per MoE layer, retaining 336 experts per layer, for an estimated 264B total parameters.

Provenance

• Observer state: /home/ubuntu/qwen397-full/observer-calibv1/qwen397-pr17-calibv1-23k-16k-observer-state.raw.pt
• Detail state: /home/ubuntu/qwen397-full/observer-calibv1/qwen397-pr17-calibv1-23k-16k-detail-state.raw.pt

Benchmarks

No benchmark summary was found.

Custom Stress

No custom stress summary was found.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("0xSero/Qwen3.5-264B", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("0xSero/Qwen3.5-264B", trust_remote_code=True)

License & citation

License inherited from the base model.

@misc{lasby2025reap,
  title  = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression},
  author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
  year   = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv}
}

Sponsors

Made possible by NVIDIA · TNG Technology · Lambda · Prime Intellect · Hot Aisle.