depth-recurrent-convergence

Depth-Recurrent Transformer (S1, step 49209)

Final recursive checkpoint for the paper Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers, and Why Reading It Beats Learning It.

A depth-recurrent (weight-tied, looped) transformer: prelude blocks, a weight-tied core looped r times with the recursion count sampled per optimization step during pretraining, then coda blocks. Trained on FineWeb-Edu for 12.9B tokens.

• Parameters: 85.6M (state dict); ~57.3M active per forward at r=1
• Training tokens: 12,900,106,240
• Step: 49209

Contents

This is an inference checkpoint (config + state_dict only; optimizer and RNG state removed). It is a torch.save dict and requires the model code from the code release to load.

import torch
from src.model import build_model

ck = torch.load("model.pt", map_location="cpu", weights_only=False)
model = build_model(ck["config"])
model.load_state_dict(ck["state_dict"])
model.eval()

# recursive forward takes a loops= kwarg
logits = model(input_ids, loops=8)

Or evaluate directly with the release harness:

python bench/benchmark.py --ckpt model.pt --tokens data/fwe-val.bin --loops 1 2 4 8 16 32
python bench/allocate.py  --ckpt model.pt --tokens data/fwe-val.bin --diagnostic

Citation

@misc{logan2026pertoken,
  title  = {Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers,
            and Why Reading It Beats Learning It},
  author = {Logan, Joe},
  year   = {2026},
  eprint = {arXiv:XXXX.XXXXX}
}