Downloads · 30 days
26
4% of all-time downloads
INTELLECT-3-57B
0xSero/INTELLECT-3-57B
INTELLECT-3-57B is a text generation model from 0xSero. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
[!TIP] Support this work → · X · GitHub · REAP paper · Cerebras REAP
All-time downloads
707
Public
Parameters
57B
114 GB on disk
Likes
3
Public
Repo makeup
Click a slice to open those files.
.safetensors114 GB · 100%
Parameter types
How the weights are stored.
BF1657B · 100%
Try a prompt
Base models
Model card
From the Hugging Face model README
[!TIP] Support this work → · X · GitHub · REAP paper · Cerebras REAP
INTELLECT-3-57B
REAP-pruned PrimeIntellect/INTELLECT-3.
At a glance
| Base model | PrimeIntellect/INTELLECT-3 |
| Format | BF16 |
| Total params | 57B |
| Active / token | — |
| Experts / layer | 64 |
| Layers | 46 |
| Hidden size | 4096 |
| Context | 131,072 |
| On-disk size | 114 GB |
50% expert-pruned version of PrimeIntellect/INTELLECT-3 using Cerebras REAP (Router-weighted Expert Activation Pruning).
Model Details
| Property | Value |
|---|---|
| Base Model | PrimeIntellect/INTELLECT-3 (248B MoE) |
| Architecture | GLM-4 MoE (glm4_moe) |
| Compression | 50% (64 experts pruned) |
| Remaining Experts | 64 per layer |
| Parameters | ~124B |
| Format | BF16 SafeTensors |
| Size | 107 GB |
REAP Configuration
dataset: 0xSero/glm47-reap-calibration-v2
samples: 1360
- evol-codealpaca-v1: 700 (code generation)
- xlam-function-calling-60k: 330 (function calling)
- SWE-smith-trajectories: 330 (agentic multi-turn)
distance_measure: angular
seed: 42
model_max_length: 2048
compression_ratio: 0.50
prune_method: reap
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"0xSero/INTELLECT-3-57B",
torch_dtype="auto",
device_map="auto",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("0xSero/INTELLECT-3-57B", trust_remote_code=True)
messages = [{"role": "user", "content": "Write a Python function to calculate fibonacci numbers"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Related Models
| Model | Compression | Format | Size |
|---|---|---|---|
| INTELLECT-3-REAP-50 | 50% | BF16 | 107GB |
| INTELLECT-3-REAP-50-W4A16 | 50% | W4A16 GPTQ | ~30GB (coming soon) |
License & citation
License inherited from the base model.
@misc{lasby2025reap,
title = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression},
author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
year = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv}
}
Sponsors
Made possible by NVIDIA · TNG Technology · Lambda · Prime Intellect · Hot Aisle.
- Task
- Text Generation
- Library
- transformers
- Type
- glm4_moe
- License
- apache-2.0
- Created
- Jan 4, 2026
- Updated
- May 30, 2026