Downloads · 30 days
15
16% of all-time downloads
looped-16x2-d640-1e18
ml-ryanlee/looped-16x2-d640-1e18
looped-16x2-d640-1e18 is a text generation model from ml-ryanlee. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
32-layer compute-optimal checkpoint for Sparse Layers are Critical to Scaling Looped Language Models (arXiv:2605.09165).
All-time downloads
94
Public
Parameters
146M
583 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors583 MB · 99%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- loop-lm
- Access
- Public
- Created
- Jul 26, 2026
- Updated
- Jul 26, 2026
- SHA
- 955307ab
Try a prompt
Model card
From the Hugging Face model README
looped-32L-d640-1e18
32-layer compute-optimal checkpoint for Sparse Layers are Critical to Scaling Looped Language Models (arXiv:2605.09165).
| architecture | looped |
| d_model | 640 |
| effective layers | 32 |
| compute budget | 1e18 FLOPs |
| training steps | 45,632 |
| parameters (stored) | 143,627,520 |
| peak LR | 0.005 |
| batch size | 16 |
| muP width_ratio | 2.5 (d_base=256) |
| final val loss | n/a |
This width is the architecture's measured minimum on a ten-rung isoFLOP sweep (d256-d1408) at 1e18 FLOPs.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
"ml-ryanlee/looped-32L-d640-1e18", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("gpt2")
Pass max_length=1024 when evaluating — the RoPE buffer is sized to the training context.
- Task
- Text Generation
- Library
- transformers
- Type
- loop-lm
- License
- apache-2.0
- Created
- Jul 26, 2026
- Updated
- Jul 26, 2026