mini-research-on-looped-transformers2026

Mini Research on Looped Transformers 2026

This model repository contains the important final 100M-token checkpoints from the research codebase. The models are small causal language models whose shared Qwen3-style decoder core is reused recurrently. The release uses raw PyTorch training checkpoints so that optimizer and experiment metadata remain available.

Released checkpoints

All runs used 99,999,744 actual training tokens, seed 42, sequence length 512, global batch 8,192 tokens, and final-output next-token cross-entropy only. The FORWARD and REVERSE curricula have identical T=8/T=16 token counts and recurrent compute; only temporal order differs.

The primary checkpoint is curriculum_8_to_16_100m.pt. Its useful inference-depth optimum moves to loop 16. The reverse control returns the optimum to loop 8.

Files required for evaluation

• src/ and scripts/evaluate.py: exact model and depth-sweep implementation.
• configs/: exact configuration for each released checkpoint.
• tokenizer/tokenizer.json: byte-level BPE tokenizer with vocabulary 16,384.
• results/: compact published metrics.
• SHA256SUMS.txt: integrity hashes for all checkpoint and tokenizer artifacts.

The FineWeb token streams are not distributed. Recreate them using scripts/prepare_data.py or follow the GitHub reproduction guide.

Download and evaluate the primary checkpoint

hf download GeneralFY/mini-research-on-looped-transformers2026 \
  --local-dir mini-research-on-looped-transformers2026
cd mini-research-on-looped-transformers2026

python -m venv .venv
source .venv/bin/activate
python -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r requirements.txt

sha256sum -c SHA256SUMS.txt

python scripts/evaluate.py \
  --config configs/curriculum_8_to_16_100m.yaml \
  --checkpoint checkpoints/curriculum_8_to_16_100m.pt \
  --data-dir /path/to/data \
  --loops 1:32 \
  --validation-tokens 65536 \
  --diagnostic-tokens 65536 \
  --kl-tokens 512 \
  --output results/reproduced_100m_65k.csv

The evaluator constructs the model from the YAML configuration and loads its state dict with strict=True. These files are research artifacts rather than a transformers.AutoModel package.

Integrity

The primary checkpoint SHA256 is:

8bc102b584df978bd7619fb752ad84ca41ed1656aa403d406f1345c92f836d6b

Tokenizer SHA256:

5f2f0950743afdc9f338fc0d0d408e4214529940bc3f9d04e0218d4ec29760f8