cl33-oplm

cl33-opLM — selective reproducibility release (preprint v1.1)

Paper: One Object: Memory, Navigation, and Reportability in an Operator-Only Language Model — DOI 10.5281/zenodo.22684392 · https://t3atlas.dev/cl33/paper/ · Live demo: https://cl33.t3atlas.dev Author: Garret Sutherland, MirrorEthic LLC.

This bundle contains what is necessary to independently test the published claims on frozen artifacts. It is deliberately not the training stack: the paper's §12 program is ongoing and its machinery is not included. Reproducibility surface ≠ complete source disclosure.

Contents

Full hashes in SHA256SUMS. Deps: torch, transformers, datasets (Python ≥3.10).

Claim 1 — the operator bottleneck is mandatory

Zero the emitted operators; the model loses its only path to output.

# EXACT in-domain reproduction (frozen 24x1025-token slice of the prose-mix val, shipped):
python repro_bottleneck.py --ckpt cl33_oplm_chat_236m.pt --slice eval_slice_prose_val.npy
# off-domain, public data:
python repro_bottleneck.py --ckpt cl33_oplm_prose_236m.pt
python repro_bottleneck.py --ckpt cl33_oplm_chat_236m.pt

Expected (WikiText-103 test, public data, seq 1024 — measured on this exact bundle):

(The paper's original 314× was a different random draw of the same validation mix; the shipped frozen slice reproduces exactly at 270×, and the claim — orders of magnitude — holds on public data too. eval_slice_prose_val.npy is derived data (GPT-2 BPE token ids) drawn from public corpora: FineWeb/FineWeb-Edu (ODC-By), DCLM, Project Gutenberg (public domain), Wikipedia (CC BY-SA), Cosmopedia (Apache-2.0), FineMath, Stack-Edu; shipped solely as an evaluation fixture with attribution.)

Claim 2 — the operator stream is a readable transcript

A probe that sees ONLY the emitted operators (no token input) decodes the text:

python repro_reverse_readout.py --text "any sentence you like"

Expected: ~0.86 top-1 on typical English (the probe's held-out card prints on load; rare words fail toward semantic neighbors — that is the paper's §7 claim, not a bug).

Training data statement

Four stages, ~11.8B tokens cumulative (GPT-2 BPE; chat stages use a 5-token spliced extension → vocab 50262). cl33_oplm_prose_236m.pt is the Stage-2 endpoint; cl33_oplm_chat_236m.pt (the cl33.t3atlas.dev demo model) is the Stage-4 endpoint.

Stage 1 — from-scratch pretrain, 5.000B tokens ("ultimate_mix"): FineWeb-Edu 2.00B (40%) · DCLM 1.00B (20%) · Cosmopedia 0.50B (10%) · FineMath 0.50B (10%) · Stack-Edu 0.50B code (10%) · Wikipedia 0.50B (10%). All 56 shards sha1-fingerprinted in the project's run manifest.

Stage 2 — context splice 512→1024 + prose continuation, 6.20B tokens ("prose_mix"): FineWeb 35% · Gutenberg 15% · Wikipedia 10% · DCLM 10% · FineWeb-Edu 10% · Cosmopedia 8% · FineMath 6% · Stack-Edu 6%. This is the checkpoint whose bits-per-byte matches token-matched Pythia-160m (paper, Table 1b).

Stage 3 — chat SFT, ~0.43B tokens, 2 epochs: 327k chat conversations (SmolTalk-derived + a deduplicated diverse set + 5,472 in-house self-Q&A pairs + 120 persona seeds), 61k self-corpus document chunks, 10k pretrain-replay documents (forgetting guard), 427 reasoning traces (287 R1-derived CoT + 140 general; system-prompt-gated).

Stage 4 — pinned-persona SFT, ~0.19B tokens: identity trained as a pinned tape record (authority channel) rather than system-prompt tokens; in-house synthetic persona corpus.

Sources are public corpora (FineWeb/FineWeb-Edu ODC-By; DCLM CC-BY-4.0; Cosmopedia Apache-2.0; Wikipedia CC-BY-SA; Gutenberg public domain; SmolTalk Apache-2.0; Stack-Edu per-repository licenses; FineMath ODC-By) plus in-house synthetic material (self-Q&A, persona, general-think; the 287 CoT traces are R1-distilled). No private or user data in any stage. The shipped eval_slice_prose_val.npy is a 24×1025-token fixture drawn from the Stage-2 validation split.

What is NOT here, and why

Training orchestration, data pipelines, the §12 memory-organ program (labeled ongoing in the paper), and downstream control/steering machinery. The claims those support are either reported with their own dated work-log provenance (paper, Appendix R) or not yet published. This bundle is scoped to verify what the preprint asserts about these frozen artifacts.

Checksums / provenance

Both checkpoints are weights-only exports (optimizer state stripped) of the exact training checkpoints named in the paper's Appendix R. Verify with: sha256sum -c SHA256SUMS