lyot-star2x5

lyot-star2x5 — latent reasoning on a branching graph

Six GPT-2-style checkpoints (2 layers, 15.0M params, 40-token symbolic vocabulary) trained with Coconut-style continuous chain-of-thought on a two-arm star, arms of length 5.

Companion to seyedparsa/lyot-chain10. Same 22 nodes and 20 edges as that depth-10 chain, so search breadth is held constant and the only thing that changes is branching: a chain has one node per depth, a star has two, and the model must carry which arm it is on. That is the setting where the logit lens showed superposition — the first thought holding both frontier nodes at 0.524/0.473.

Task

Which of two candidate nodes is reachable from the root?

<eos> 0 1 | 2 0 | 1 6 | 5 3 | 4 7 | 7 5 [Q] 3 6 [R] 2 <L> <L> <L> [A]  ->  6
      \_______ edges _______/            \cands/  \root/  \thoughts/

The distractor is the target's counterpart in an isomorphic unreachable twin — same depth, same in/out degree — so no local feature separates the candidates. Node ids are permuted per task.

Chance on this benchmark is 0.500, since two candidates are offered. Chance on an intermediate curriculum stage is 0.045: those prompts still show the final pair, so the intermediate node must be generated from the whole 22-node vocabulary. Conflating the two is what produced the correction below.

Checkpoints

Each is a full training state: model, optimizer, epoch, stage, metrics.

eval/acc is on 1000 held-out graphs at full difficulty and full latent budget. Filenames embed the wandb run id (project lyot-star2x5). All arms share the recipe that solved chain-10: uniform_prob 0.5, reset_optimizer true, shuffle_nodes false, 2 layers.

Findings

Ring supervision beats path, decisively. s2x5r-adaptive reaches 0.999 where the same configuration with curriculum: path never leaves stage 0 in 504 epochs (0.025). On a star the path curriculum supervises one specific arm's node at each depth — a coin flip before the model knows which arm the target is on — while the ring supervises the frontier, which is well defined without that knowledge. Both clock arms sit at chance regardless of curriculum.

Correction (2026-07-27): an earlier version of this card said the 0.991 arm "never learned a single intermediate stage". That was wrong. It read per-stage scores of ~0.5 as chance, when chance here is 0.045 and ~0.5 is the ceiling.

Why: the prompt shows the final target pair, so an intermediate node has to be generated from the whole 22-node vocabulary rather than selected from two options. And on a 2-arm star the ring holds exactly two nodes, one of which is sampled as the label each epoch — so a model that has learned the frontier perfectly still scores ~0.5 on exact match, because it must pick one member and the other was the label half the time.

Probing s2x5r-adaptive's own checkpoint, scoring each stage at its own budget:

117–120 of 120 at every stage. It had learned every intermediate stage essentially perfectly, and its 0.45 threshold was correctly calibrated for a metric whose maximum is 0.5 — not a leaky gate. The ranking in the table above is real.

Caveats

• The set-loss comparison is still unresolved. s2x5rset-adaptive (0.765) and s2x5rgate-adaptive (0.800) differ by less than noise, and both were stopped ~60 epochs younger than the arm they are compared against.
• Snapshots, not converged or best-ever weights. All runs were cancelled manually; no LR decay is implemented.
• Trained on a synthetic 40-token symbolic vocabulary; not a language model.

Loading

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download("seyedparsa/lyot-star2x5",
                       "s2x5r-adaptive-lyot-star2x5_zojwxj05-ep189-s5-eval0.991.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=False)
print(ckpt["epoch"], ckpt["stage"], ckpt["metrics"]["eval/acc"])
state = ckpt["model"]

Config: n_layer=2, n_head=8, n_embd=768, n_positions=1024, method=coconut, c_thought=1, max_latent_stage=5, lr=1e-4, batch_size=128.