Boris-2-0917

Boris-2-0917 (aborted pipeline revision)

This is a weights-only snapshot from the Boris-2-0917 run at 7.107B tokens (step 13,556 of a planned 200B). It is published as an honest record of an aborted experiment, not as a finished or recommended model.

The model has 143,724,032 parameters: 31 layers, width 512, full 8-head MHA, SwiGLU width 1664, a 15M-parameter explicit n-gram table, and a tied 32,768-token byte-level BPE. The parameter increase over 0907 is 0.15% and came from rebalancing the fixed budget toward attention.

What failed and what did not

The weights were numerically healthy when archived: loss was approximately 2.98, the loss scale was stable at 32,768, and gradient norms were finite. The run was stopped because a full audit found pipeline choices that should not be carried through another two months of training:

1. The 32k tokenizer was custom, but it had been trained for the previous data mixture rather than ClimbMix. A ClimbMix-native 32k BPE encodes held-out ClimbMix about 0.89% more efficiently, equivalent to roughly 1.8B tokens of additional text over a 200B-token budget.
2. The n-gram table allocated 60% of its rows to bigrams and 40% to trigrams. Held-out testing found a roughly 90/10 allocation stronger at the same table size. Quadgrams and quintgrams contributed too little to justify rows.
3. Documents longer than 65,536 tokens were truncated rather than continued in a later packed row. Across the prepared corpus, about 882.5M stored tokens were unreachable.
4. The prefetcher checkpointed producer state ahead of the last consumed batch. A resume or context-stage transition could therefore skip prefetched data.
5. The local multiple-choice evaluator assumed BPE prefix stability and used incorrect HellaSwag text formatting. The historical scores below should be treated as development diagnostics rather than leaderboard-comparable results.
6. Separate GradScaler decisions could, on an overflow, update Muon parameters while skipping AdamW parameters. No such persistent overflow was observed in this snapshot, but the behavior was unsafe for a long run.

These issues were corrected for Boris-2-0920 without increasing model size.

Historical 7B evaluation

The run's original local evaluator reported:

Mean: 39.81. Because of the evaluator problems described above, these numbers are included for provenance and should not be used as a clean model comparison.

Files

• model.pt: fp16 weights plus serialized configuration, at 7.107B tokens.
• tokenizer/: the exact byte-level BPE used by this checkpoint.
• ngram_vocab.bin: the exact n-gram-to-row mapping used during training.
• bench_7B.log and train_20260920_124514.log: historical evaluation and training logs.
• code/: custom architecture and checkpoint utilities.

The checkpoint is a custom PyTorch format and is not directly loadable with AutoModelForCausalLM. Do not replace its tokenizer or n-gram vocabulary: the tensor shapes would still match while learned row meanings would be wrong.