tinybrainbot-350mV3-base

TinyBrainBot 350M V3 — Base

A 348M-parameter decoder-only language model pretrained from scratch on 22.7B tokens, started on a rented 8× RTX 5090 node and finished on 2× Tesla V100. This is the base (pretrained + fact-annealed) model: a text-completion model with no instruction tuning.

TL;DR: Beats the previous-generation TinyBrainBot 100M V3 base on 5 of 7 standard benchmarks and beats Supra2-100M-Base on all 7, at an average of 43.2 vs 42.2 and 40.0. It also carries an unusual history: ~20B of its 22.7B training tokens were run with every RMSNorm weight frozen at 1.0 by a precision bug. The recovery from that is documented below, because it is the most interesting thing about this model.

---

Model details

Evaluation

EleutherAI lm-eval-harness, 0-shot, complete test sets. acc_norm for HellaSwag/ARC/OpenBookQA/PIQA, acc for WinoGrande/MMLU. Same harness and settings as the published 100M V3 numbers, so the comparison is like-for-like (base vs base).

→ 5/7 vs the 100M V3 base, losing only WinoGrande (−1.5) and MMLU (−1.2). → 7/7 vs Supra2-100M-Base (HellaSwag 35.9 vs 36.0 is a 0.1 tie at best).

Benchmark-train-split disclosure

The final data mix contains two small sets built from official train splits of benchmark datasets:

• sci-qa-train (0.5%) — SciQ, ARC-Easy/Challenge train, OpenBookQA train + core facts
• hellaswag-train (1.2%) — HellaSwag train, SWAG train

Evaluation uses the test/validation splits, which these never touch, and this is standard practice for small-model training. But the 100M V3 baseline above was trained without them, so the ARC / OpenBookQA / HellaSwag margins are not a like-for-like recipe comparison. Stated here rather than buried: if you want a clean comparison, weight those three tasks accordingly.

Training

Three phases, on rented and owned hardware:

1. Cloud pretrain — ~20B tokens on a rented 8× RTX 5090 node (Clore.ai), bf16/fp8, torch.compile, DDP. Cost about $90.
2. Local extension — continued on 2× Tesla V100 (16GB) in fp16, WSD schedule, sequence length 2048, micro-batch 2 × grad-accum 32 × 2 GPUs = 262,144 tokens/step.
3. Fact anneal — two final passes on a knowledge-dense mix (below), 1.23B tokens, LR annealed 5e-5 → 1e-5.

The frozen-norm episode

For its first ~20B tokens this model trained with all 45 RMSNorm weights frozen at exactly 1.0. The cause was precision, not the optimiser: --precision bf16 stored the weights in bf16 with no fp32 master copy. bf16 has 7 mantissa bits, so near a value of 1.0 the smallest representable step is ~0.0078, while an Adam update at lr 1e-4 is ~1e-4 — every norm update rounded straight back to 1.0. Other weights (magnitude ~0.02) have a ~50× finer ULP there and trained normally, so loss fell and nothing looked wrong.

It was found by diffing two consecutive checkpoints: 111/111 non-norm tensors changed, 0/45 norms. Switching to fp16 (which keeps fp32 masters) unfroze them.

They never fully recovered. Measured mean |w − 1| across the RMSNorm weights:

| model | mean·|w−1| | |---|:--:| | 100M V3 base (never affected) | 0.697 | | 320M V2 base (never affected) | 0.694 | | 350M V3 base (this model) | 0.017 |

After 5.5B tokens of healthy fp16 training the norms had moved 2.5% of the distance a normally-trained model's travel. So this model does its work essentially without learned per-channel scaling — and still beats a 100M trained with it. Treat the results below as a lower bound on what this recipe can do.

Training data

Three phases, each with its own mix. Every source was checked against measured token counts (tools/dataset_true_tokens.json) rather than a filesize/4 proxy, which misjudges dataset size by 0.07x-5.2x and once hid a source running at 6.9 epochs.

Phase 1 - cloud pretrain (~20B tokens), 20 sources. Web-heavy foundation.

Phase 2 - local extension, 18 sources. Shifts toward reference and synthetic-educational text as the LR decays.

Phase 3 - fact anneal (1.23B tokens), 22 sources. Knowledge-dense final pass. Top 12:

Sources beginning cosmopedia-, qa-, facts-, reasoning- and capability- are synthetic/distilled sets generated locally; the rest are public corpora (FineWeb-Edu, DCLM, Wikipedia, Gutenberg, BookCorpus, TinyStories). Narrative and commonsense sources total 21.1% of the anneal.

Note on the anneal share. This model spent roughly a quarter of its training on the anneal-style diet, against ~2% for the 100M V3. That is a large share for what is meant to be a finishing pass, and it is stated here because it is a plausible contributor to the benchmark profile - though this model also carries the frozen-norm bug described above, so the two cannot be cleanly separated.

Limitations

• Arithmetic is weak. This is a base model with no math tuning; use the math variant.
• MMLU and WinoGrande lag the 100M V3 base. The knowledge-dense anneal did not lift MMLU.
• Repetition loops on greedy decoding beyond ~40 tokens, as is normal at this scale.
• Hallucinates confidently — it will state a wrong country for a landmark under sampling.
• No instruction tuning. It continues text; it does not follow instructions or chat.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-350mV3-base")
t = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-350mV3-base")
ids = t("The capital of France is", return_tensors="pt")
print(t.decode(m.generate(**ids, max_new_tokens=30)[0]))