Boris-2-0907

Boris-2-0907 (superseded research checkpoint)

This is the final weights-only export from a Boris-2 run stopped at 30.00B tokens (step 57,221). It is published as a research artifact and negative result, not as a finished or recommended language model.

The model has 143,504,896 parameters: 26 layers, width 512, 8 query heads / 2 KV heads, SwiGLU width 2368, a 15M-parameter explicit n-gram table, and a tied 32,768-token byte-level BPE.

Why the run was stopped

Three design choices made continuing the planned 200B-token run a poor use of compute:

1. AdamW weight decay was accidentally zero for the embedding and n-gram table. Their norms grew 21.1% between 20.8B and 30.1B tokens while the Muon-controlled matrices remained within 1%. The effective learning rate on embedding directions was steadily shrinking. Weight decay was corrected near the end, but most of this checkpoint had already trained under the bad setting.
2. The model devoted only 17.0M parameters to attention and used 4:1 GQA even though the experiment did not need a small inference KV cache. Its largest benchmark deficits were on PIQA and HellaSwag, both sensitive to contextual integration.
3. The initial data mixture contained 30% synthetic or structured text from token zero. Held-out FineWeb-Edu loss was worse than loss on the model's own mixture, indicating insufficient broad natural text.

The run did not fail because of a corrupt checkpoint. It was deliberately superseded after these problems became measurable.

Historical evaluation

Zero-shot acc_norm, lm-eval 0.4.12:

Intelligence Index: 15.89. These are intermediate-run measurements, not a claim of parity with a fully trained model.

Files

• model.pt: fp16 weights plus serialized training configuration, at 30.00B tokens.
• tokenizer/: the exact byte-level BPE used by the checkpoint.
• ngram_vocab.bin: the exact n-gram-to-row mapping used during training.
• config_training.py: archived configuration from the run.
• results.jsonl, metrics.jsonl, train.log: evaluation and training history.
• code/: loader/model components needed to inspect the custom architecture.

The checkpoint is a custom PyTorch format and is not directly loadable with AutoModelForCausalLM. The tokenizer and n-gram mapping are inseparable from the learned weights; substituting either one assigns different meanings to learned rows.