lyot-chain10

lyot-chain10 — Coconut latent reasoning on a depth-10 chain

Four GPT-2-style checkpoints (2 layers, 15.0M params, 40-token symbolic vocabulary) trained with Coconut-style continuous chain-of-thought on a graph-reachability task requiring ten sequential latent steps.

Part of Learn Your Own Thoughts (ATC). Coconut's previous ceiling in our experiments was depth 4 under a fixed clock schedule; these reach eval accuracy 0.999 at depth 10.

Task

Which of two candidate nodes is reachable from the root?

<eos> 3 17 | 8 1 | 15 16 | ... | 5 7 [Q] 19 13 [R] 20 [A]  ->  19
      \___ edges ___/               \cands/  \root/

A depth-10 chain plus an isomorphic unreachable twin — 22 nodes, 20 edges, 77 tokens. The distractor is the target's counterpart in the twin component, matching depth and in/out-degree, so no local feature separates the candidates. Chance is 0.500. Node ids are permuted per task, which removes an id-ordering shortcut worth ~0.30 accuracy.

Checkpoints

Each is a full training state: model, optimizer, epoch, stage, metrics.

eval/acc is on 1000 held-out graphs at full difficulty and full latent budget. The filename embeds the wandb run id (project lyot-chain10). All four were at curriculum stage 10 when saved. The best is shuffle_nodes=true, the shortcut-free setting.

The clock baseline (c10clk-*), at chance

The four c10clk-* files are the control, and they are chance-level by design, not broken. Everything about them matches the arms above except that the curriculum advances on upstream's fixed schedule (epochs_per_stage: 5) rather than on measured accuracy:

Chance is 0.500. All four reached stage 10 by epoch 50 — three times sooner than the adaptive arms — and none exceeded 0.537 at any point in ~160 epochs. They are published because the depth-10 claim rests on this comparison: same task, same model, same reset_optimizer=true, differing only in how stages advance.

What made depth 10 reachable

Two changes over the standard recipe, from a 2x2x2x2 factorial plus this baseline:

1. reset_optimizer=true — rebuild AdamW every epoch, discarding both moment estimates. Necessary: all four adaptive arms with it reached stage 10 and >0.96 eval; no arm without it passed stage 7 or 0.07 eval, given ~2x the epochs. The curriculum keeps shifting the input distribution, and stale second moments appear to misscale the new stage's gradients.
2. Accuracy-driven staging (staging: revisit, threshold 0.9, patience 5) — advance only after a stage passes 5 consecutive epochs at its own budget, and fall back to any earlier stage that regresses.

Neither alone is enough. The c10clk-* arms have the optimizer reset and still sit at chance, so the reset is necessary but not sufficient; and no arm without the reset got near stage 10 under either schedule. Arriving at the final stage early is worth nothing if the stages were never learned on the way.

uniform_prob and shuffle_nodes only modulate how fast the schedule progresses; neither rescues a run without reset_optimizer.

Caveats

• Snapshots of a live oscillation, not converged models. The revisit scheduler repeatedly drops back to an earlier stage, and eval swings between ~0.3 and 1.000 before recovering. These are caught at stage 10; they are not best-ever weights, and no LR decay was used.
• Depth 10, not depth 10 held. The 1.000 recurs across many epochs rather than once, but "solves chain-10" describes a peak becoming a plateau.
• 2 layers beats 4. In the same factorial no 4-layer arm passed stage 7 (best eval 0.17).
• Trained on a synthetic 40-token symbolic vocabulary; not a language model.

Loading

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download("seyedparsa/lyot-chain10",
                       "c10-L2-u0.5-rtrue-strue-lyot-chain10_ucou7ar1-ep157-s10-eval0.999.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=False)
print(ckpt["epoch"], ckpt["stage"], ckpt["metrics"]["eval/acc"])
state = ckpt["model"]

Config: n_layer=2, n_head=8, n_embd=768, n_positions=1024, method=coconut, curriculum=path, c_thought=1, max_latent_stage=10, lr=1e-4, batch_size=128.