Downloads · 30 days
12
16% of all-time downloads
looped-moe-16x2-d512-1e18
ml-ryanlee/looped-moe-16x2-d512-1e18
looped-moe-16x2-d512-1e18 is a text generation model from ml-ryanlee. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
32-layer compute-optimal checkpoint for Sparse Layers are Critical to Scaling Looped Language Models (arXiv:2605.09165).
All-time downloads
74
Public
Parameters
209M
835 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors835 MB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- loop-lm
- Access
- Public
- Created
- Jul 26, 2026
- Updated
- Jul 26, 2026
- SHA
- 0df955fb
Try a prompt
Model card
From the Hugging Face model README
looped-moe-32L-d512-1e18
32-layer compute-optimal checkpoint for Sparse Layers are Critical to Scaling Looped Language Models (arXiv:2605.09165).
| architecture | looped-moe |
| d_model | 512 |
| effective layers | 32 |
| compute budget | 1e18 FLOPs |
| training steps | 65,960 |
| parameters (stored) | 206,717,952 |
| peak LR | 0.005 |
| batch size | 16 |
| muP width_ratio | 2.0 (d_base=256) |
| final val loss | n/a |
This width is the architecture's measured minimum on a ten-rung isoFLOP sweep (d256-d1408) at 1e18 FLOPs.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
"ml-ryanlee/looped-moe-32L-d512-1e18", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("gpt2")
Pass max_length=1024 when evaluating — the RoPE buffer is sized to the training context.
- Task
- Text Generation
- Library
- transformers
- Type
- loop-lm
- License
- apache-2.0
- Created
- Jul 26, 2026
- Updated
- Jul 26, 2026