bind1-babylm2026-strict-small

bind1 — Binding-Loop LM (24M), BabyLM 2026 Strict-Small entry

bind1 is the 2026 competition generation of a planned architecture ladder built around explicit role binding; the accompanying paper describes it as a microkernel loop LM. Its successor, bind3 — a governed-iteration design whose authority chain learns when to trust the iterative pathway (whose original backup-pathway motivation did not survive the revised metric — see the honest notes below; the 2027 question is now stated against the leakage-free standard) — is planned as our BabyLM 2027 entry. Both generations follow the same policy: present the most honest, most complete account of what the mechanism does and does not do. A 23.9M-parameter causal language model trained from scratch on the official BabyLM 2026 Strict-Small corpus (~10M words), within the 10-epoch exposure cap (150M tokens ≈ 9.2 epochs). The architecture is a weight-tied binding loop: an input block (3 layers), a core block (4 layers) unrolled T=3 times with an explicit role-binding write into a reserved 64-dim slice per iteration, and an output block (3 layers) — effective depth 18. A learned trust signal (v_gain) couples the model's self-consistency estimate to how much each iteration's re-binding is trusted during training.

• Entry id (internal): bind1_tt3_s0 (physical run name loop2_full), random seed 0 (BABYLM_SEED=0), frozen 2026-07-05 before any replication seeds were trained (no post-hoc seed selection).
• Track: Strict-Small. Backend: causal.

Architecture at a glance

input ─▶ [labeling]  bind a soft, revisable logic-role to each term
              │       (written into a dedicated 64-dim role slice; re-stamped
              │        each iteration, never accumulated; content untouched)
              ▼
         [inference]  one weight-tied core pass — no bypass path, by design
              ▼
        [consistency] a per-position verdict v on the result
              ▼
         [volition]   next iteration's binding TRUST hears that verdict
              │        through a learned gain + per-role experience priors
              └──────▶ re-bind … (×T)  ─▶  next-token logits

Grammar competence lives partly in the iterations (+3–4.5 BLiMP, 4/4 seeds); the entity-tracking effect lives in the binding write itself — formed during training because of the loop, harvestable in a single pass at inference (as measured by the full-item-set metric; see the metric-revision section below for what the revised leaderboard standard changes about this reading). Every pathway is instrumented (see train_loop2_full.json), and the audit script can tell you whether any exported checkpoint's loop is live or bypassed.

Why this entry might matter beyond its scores

Three things in this work are, we believe, worth scrutiny independent of leaderboard position (each stated with its honest scope: 24M params, one benchmark family, one architecture):

1. An inversion of the test-time-compute arrow. In our data, iterating during training is what makes the binding capability form at all — yet at inference the capability is harvestable in a single pass (the trained-depth sweep and the from-scratch controls in the companion papers and the ablations repo carry the full evidence). If this train-iteratively-deploy-single-pass pattern survives larger scales, it is a compute-economics counterpoint to scaling inference-time compute: buy the capability once at training time, serve it cheap. We state it as a falsifiable hypothesis, not a claim.
2. A guard against self-deception in adaptive-compute evaluation. Best-of-$T$ "oracle headroom" — the standard ceiling reported for adaptive computation — can be pure re-rolled noise. Our chance-null protocol (compare conditional recovery against the $1/(k{-}1)$ re-roll baseline) returned both verdicts in this very repo's data: no exploitable signal in three healthy seeds, genuinely above-chance recovery in the collapsed one. Our own +8-point "headroom" died by this test before the real signal was found. We think any test-time-compute result reporting best-of-$N$ ceilings benefits from this control.
3. The binding write, and exactly what is new about it. Role-filler binding is a classic problem (TPR; TP-Transformer; SysBinder) and our write primitive is borrowed (NTM/delta-rule) — all cited. What we found no essential precedent for, after an adversarial prior-art search, is the wiring: a consistency verdict adjusting next-iteration binding trust through a learned gain plus per-role experience priors — and the empirical result that the write itself (not iterative re-binding) carries a 4/4-seed paired entity-tracking gain (full-item-set metric; see the metric-revision section below), on an instrumented pathway that lets us audit when it works, when it collapses, and when it is decorative.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("REPO_ID", trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained("REPO_ID")

The model is fully self-contained (modeling_babylm.py ships in the repo); forward() runs the whole loop and returns standard CausalLMOutput(logits, loss). Empty-context, stateless across examples (competition-compliant: no external memory is used in training or evaluation).

Results (official babylm-eval, zero-shot, final checkpoint)

GLUE fine-tuning (official protocol): BoolQ 67.9 · MNLI 45.1 · MRPC-F1 82.2 · QQP-F1 62.9 · RTE 54.7 · MultiRC collapsed to the majority class (F1 0.03 — shared with our size-matched monolith control; see paper for the tokenizer-separator analysis).

Two entity-tracking numbers: the 2026-07-07 metric revision

On 2026-07-07 the official leaderboard revised its entity-tracking standard (leaderboard commit a010bbce, "removal of nothing answers"): items whose gold completion contains "nothing" are excluded from scoring (per-subtask expected counts drop from the 607-series to the 508-series), and a prediction/gold alignment bug was fixed in the same commit. The public evaluation pipeline available to participants (babylm-eval commit 02b56cb, 2026-06-28, with the released BabyLM-2026-Strict-Evals data) scores the full item set — that is the standard under which this card's 39.55 (and every entity number in the companion paper) was computed.

Why the removal is sound (our own audit of the released data agrees with it): across all 9,483 entity items, "nothing." never appears as a distractor — whenever it is among the five options, it is the gold answer, so its presence leaks the answer; the unnormalized sum-of-log-prob ranking additionally favors it as the shortest option. About 28.5% of items have a "nothing" gold.

How the 19.22 was produced — no retraining, no new model run. It is a direct re-scoring of the same frozen checkpoint's already-published per-item predictions: the gold-contains-"nothing" items are filtered out exactly as the revised leaderboard rule prescribes, and the resulting macro score matches the public leaderboard display for this entry exactly. The weights, the predictions, and every other number on this card are unchanged. All per-item predictions needed to recompute both numbers yourself are in the companion eval-artifacts dataset.

Leaderboard comparability note. The revision applies to submissions going forward: the leaderboard reads entity predictions only under the new entity_tracking_filtered key (leaderboard commit cd1994a, 2026-07-08, explicitly with no fallback to the old key), and submissions must be re-collated and resubmitted for entity to be scored under the revised standard. Entries evaluated before the revision may therefore still display full-item-set entity scores until their authors resubmit — the two standards can coexist on the board for a while, so entity columns are not always directly comparable across entries. This entry's displayed 19.22 follows the revised standard.

Honesty note. Under the revised standard, this model and our size-matched monolith control both land at ≈ the five-way chance level (19–20), in all four seeds (paired loop-minus-monolith difference: −0.4±0.4 evaluated single-pass, +0.5±0.5 at the entry's trained depth). The four-seed paired entity-tracking advantage reported on this card and in the companion paper (+12.7±5.7 at T=1) is carried entirely by the nothing-gold items — it is a full-item-set-metric result. The ablation structure behind it (the ten-seed necessity test, the frozen-edge control, the severing test) is unaffected as a description of what the training mechanism installs; under the revised standard, that effect simply no longer contributes to the score.

Growth checkpoints (competition requirement)

Branches chck_1M … chck_10M (every ~1M words through the first epoch) and chck_20M … chck_90M (every ~10M words thereafter; training ends at ≈92M words, so chck_100M does not exist). Checkpoints were saved on a fixed token cadence (1.63M tokens ≈ 1M words; 16M tokens ≈ 9.8M words), so nominal branch labels deviate from exact word counts by ≤1.7% — the mapping table:

Studying the learning curve (checkpoint branches)

The 18 growth checkpoints double as a learning-dynamics resource — each branch is a fully loadable model, so questions like "when does entity tracking emerge?" take one line per point:

model = AutoModelForCausalLM.from_pretrained(
    "SecludedCorner/bind1-babylm2026-strict-small",
    revision="chck_5M",   # any branch from the table above
    trust_remote_code=True)

The first-epoch cadence (every ~1M words, chck_1M…chck_10M) is deliberately dense for early-emergence studies; later checkpoints follow every ~10M words.

Full evaluation artifacts (companion dataset)

Every number above — and every ablation family behind the honest notes — is downloadable and recomputable: per-item predictions (with correctness fields), official reports, and the analysis scripts live in the companion dataset repo SecludedCorner/bind1-babylm2026-eval-artifacts.

Training telemetry & audit tooling

• train_loop2_full.json — the complete training curve (1.8k records): loss/ppl plus the loop's full mechanism telemetry (verdict-to-trust gain v_gain, trust τ, verdict v, role entropy, per-iteration binding revision dBind, per-role priors). Every pathway of the loop is instrumented; this file is the primary evidence that the published mechanism was live during training (and it is how a silently-bypassed earlier variant was caught).
• audit_gate_bypass.py — a standalone audit script: point it at any exported model directory (python audit_gate_bypass.py <model_dir>) to check whether the loop actually contributes to the forward pass or has collapsed to a bypass. Reproduces the audit reported in the accompanying paper; runs on CPU in seconds.

Reproducibility

Full training / export / audit / analysis code: github.com/SecludedCorner/bind1-babylm2026 (MIT).

Trained on a single laptop GPU (RTX 4060, 8GB) in ~2h:

set BABYLM_SEED=0
python train_loop.py 384 3 4 3 3 256 16 150000000 loop2_full 16000000

(dim 384, in 3, core 4, out 3, T 3, seq 256, batch 16, 150M tokens, checkpoint every 16M tokens; first epoch additionally checkpointed every 1.63M tokens.) Tokenizer: 16k BPE trained on the Strict-Small corpus only. Full reproduction guide accompanies the paper.

Honest notes (the complete picture, including negatives)

• The submitted checkpoint is seed 0; the science is four-seed. Competition rules score one checkpoint — ours was frozen 2026-07-05, before any replication was trained (no post-hoc seed selection; in hindsight it is also the strongest of the four on entity tracking, but no claim depends on that). Every headline claim is replicated across four random seeds with paired statistics: the single-pass ($T{=}1$) binding write beats the same-seed monolith on entity tracking in 4/4 (paired +12.7±5.7 — full-item-set metric; under the 2026-07-07 revised leaderboard standard this advantage is absent, see the metric-revision section), BLiMP stays above the GPT-2 baseline in 4/4 (66.1±0.4). Seed variance is large and one seed's write collapsed outright; full tables in the paper, nothing hidden. The replication checkpoints are in this repo (branches seed1/seed2/seed3) so the paired claims are verifiable end-to-end.
• What inference-time iteration does is two-sided, and we report both sides. It helps grammar in every seed (BLiMP +3.2 to +4.5 over single-pass, 4/4 — one reason the entry runs at $T{=}3$). On entity tracking it erodes the healthy write (3/3 seeds, monotonically), yet in the one seed whose write collapsed it partially rebuilds tracking — and a per-item chance-null analysis reads that recovery as above the 25% re-roll null (37–45%) under the pre-revision item set. The same analysis shows no exploitable per-item headroom in the healthy-write seeds. Post-revision update (2026-07-09): re-scored under the leaderboard's revised entity-tracking standard (removal of "nothing"-gold items, commit a010bbce), the collapsed-write seed's recovery rate falls to 23.4–24.7% ≈ the 25% re-roll null, and the recovered items are 60–70% "nothing"-gold — the recovery restores the expression of a state-blind completion prior, not tracking. We retire the backup-pathway reading and keep the chance-null protocol, which is what caught it (in both directions, twice).
• The trust edge (v_gain) is causally inert at inference in all four seeds, but training-relevant: frozen from scratch, the single-pass advantage fails to form in three of four seeds (and formed anyway in the fourth) — a training scaffold that raises the odds the write forms, not a strict necessity.
• Learning when iteration carries signal at all — now judged against the revised, leakage-free standard, under which the 2026 recovery signature did not survive — is the core mechanism question the 2027 generation (bind3) must answer before it earns a governed-iteration design.

Citation

Yulin Yang (ORCID 0009-0007-4827-8449). A Microkernel Language Model: Reasoning as Mutually-Supporting Aggregates, and Why Its Parts Must Be Judged Together. BabyLM Challenge 2026 (Strict-Small track) submission.