webLTM

webLTM — a 153K-parameter recurrent-depth transformer

Answers addition problems by looping a single transformer block over its own latent state r times, then reading out every output digit at once. There is no chain-of-thought text channel in this architecture — nothing that happens inside the loop is ever decoded into a token. r is supplied at inference time, not fixed by training, so one checkpoint answers at any thinking depth.

This is a small, fully open, from-scratch build of the mechanism behind what OpenAI called "opaque recurrence" in Astra (Sept 2026) and behind the recurrent-depth / looped-transformer line of research — most directly Geiping et al., "Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach" (arXiv:2502.05171), whose random-recurrence-depth training trick this model uses.

Code + browser demo + full write-up: see the webLTM GitHub repo (train.py, eval scripts, and a live in-browser demo with an interactive thinking-depth slider).

Architecture

tokens → embed → prelude (1×) → core (×r, looped) → coda (1×) → head → digits
                                  ▲        │
                                  └────────┘  prelude output re-injected every iteration

• prelude, core, coda are pre-LN transformer blocks (multi-head attention + GELU MLP). core is applied r times with the prelude's output re-injected at every step (h = core(h + e0)), which keeps the latent state from drifting as r grows.
• d_model=64, 4 heads, 152,909 parameters total, 615KB in float32 safetensors.
• Task: 5-digit + 5-digit addition, all output digits predicted simultaneously (non-autoregressive) — no scratchpad, no per-token decode loop.
• Trained with r sampled uniformly from [1, 8] on every step, so the weights aren't specialized to one fixed depth.

Usage

# pip install torch safetensors
from modeling_webltm import load_webltm, tokenize, decode

model, cfg = load_webltm("model.safetensors", "config.json")
ids = tokenize(cfg, 4821, 367)

import torch
with torch.no_grad():
    logits = model(torch.tensor([ids]), r=8)   # r = thinking depth
digits = logits.argmax(-1)[0, cfg["ans_start"]:].tolist()
print(decode(cfg, digits))   # -> "005188"

modeling_webltm.py (included in this repo) is the entire model definition — no trust_remote_code framework dependency, just plain torch.nn.

Results: accuracy by loop depth and problem difficulty

Exact-match accuracy, evaluated on held-out problems (300 per cell):

Harder (more-digit) problems keep improving with more loops through the trained range; pushed well past it (r=16, r=24) accuracy collapses — the re-injection trick stabilizes the recurrence within the trained range, it doesn't make the coda's read-out valid for latent states it never saw during training. That ceiling is disclosed here deliberately, not hidden.

Scope

Not a reproduction of OpenAI's Astra — Astra is an undisclosed, frontier-scale system, and the actual controversy around it is about chain-of-thought monitorability disappearing at a scale where a lot could go wrong unseen. This is 153K parameters doing arithmetic, built at a scale small enough to read every line of the loop — the opposite move from "opaque," on purpose.

Files

• model.safetensors — trained weights (float32, no pickle)
• config.json — architecture + tokenizer config
• modeling_webltm.py — full model definition (copy-paste runnable)