tinybrainbot-320mV2-base

TinyBrainBot 320M V2 — Base

A ~326M-parameter decoder-only language model pretrained from scratch on ~10B tokens on 2× NVIDIA Tesla V100 (16GB). This is the base (pretrained + quality-annealed) model — a text-completion model with no instruction tuning.

• For the chat / instruction-following version, see tinybrainbot-320mV2-instruct.
• Successor to the 303M V2 base.

TL;DR: A compact general-purpose base LM trained from scratch on ~10B tokens. It matches/beats Pythia-410M on general benchmarks (on far fewer training tokens) and cleanly sweeps GPT-2-124M. Its 2–3-digit arithmetic is notably strong for the parameter count (94–99% on 2–3-digit addition) — a product of curated numerical pretraining data — though it hits a clear length-generalization wall beyond 3 digits and is not math-specialized.

---

Model details

Usage

This is a base completion model — it continues text and does not follow a chat template. Prompt it with text to continue:

prompt = "The capital of France is"
# -> the model continues the text

For Q&A / chat behavior, use the instruct variant.

Format: provided as fp16 safetensors (AutoModelForCausalLM). No GGUF is shipped for the base model — it's a raw completion model, and GGUF chat apps expect an instruction-tuned model. For chat / GGUF, use the instruct variant.

---

Training procedure

Pretraining loss over 51,000 steps / ~10B tokens — the dip near step 40k is the quality-anneal phase.

Trained with the custom TinyBrainBot trainer using PyTorch DDP (gloo backend) across 2× Tesla V100-PCIE-16GB on Windows, fp16 autocast + GradScaler, fused AdamW. One continuous Warmup–Stable–Decay (WSD) run.

Two-phase data: a broad base phase (0 → ~40k steps) followed by a quality-anneal phase (~40k → 51k) in which the data mix was swapped to a knowledge-dense blend (Wikipedia leads up, web data halved, math tripled) as the LR decayed. The anneal pulled loss from ~1.62 to 1.436.

Pretraining data mix (sampling weights; design principle: real > synthetic, synthetic ≤ ~35%):

(weights are sampling proportions; sum = 9.35. Explicit math ≈ 2.1% of the mixture — but ~215M tokens in absolute terms, the bulk of the model's math exposure.)

---

Evaluation

Measured on our own log-likelihood MC harness (lm-eval style, fixed seed). Headline metric = acc_norm for HellaSwag/ARC/OpenBookQA, acc for WinoGrande/MMLU. Reference values are published lm-eval approximations — treat gaps under ~±2 points as ties.

vs the previous 303M base (n=2000)

→ Clean 6/6 sweep over the previous generation.

vs reference models (headline metric)

→ Sweeps GPT-2-124M 6/6; ~5 wins + 1 draw vs Pythia-410M. SmolLM-360M (trained on ~600B aggressively-filtered tokens) remains the out-of-reach frontier for this size.

Training efficiency. These results come from ~10B pretraining tokens — roughly an order of magnitude fewer than the Pythia suite's ~300B. The Pythia-410M parity is best read as a token-efficiency result (curated data + quality anneal) rather than a scale win.

Arithmetic (GPT-3 Arithmetic protocol, exact-match, n=300/sub-task)

→ Strong through 3 digits, then a hard wall. 2–3-digit addition is near-solved (94–99%, with correct carrying), but 4+-digit accuracy collapses to ~0%: the model executes a fixed ~3-column addition routine and silently drops the higher place values — a length-generalization limit tied to the training distribution (math-v2 operands are ≤3 digits), not truncation (generations complete normally). Subtraction sits ~42–50% — it handles a−b when a>b but drops the sign on negative results. 2-digit multiplication ~21%; single-digit composite ~4%.

How to read this: the GPT-3 Arithmetic suite mainly probes exact symbolic computation and short-range algorithmic generalization; it should not be read as evidence of mathematical reasoning (word problems), which is a separate, much harder skill at this scale.

---

Intended use & limitations

Intended use: research on small-model training, base for further fine-tuning, text completion, and arithmetic computation.

Limitations:

• Base completion model — does not follow instructions or a chat format (use the instruct variant for that).
• Math reasoning (word problems) is at the floor — the model computes but does not reason through multi-step problems.
• Negative-result subtraction is unreliable (drops the sign).
• WinoGrande and MMLU sit near the random floor — consistent with the capacity and data limits of a ~326M model under this training recipe.
• Trained predominantly on English; 1024-token context; no safety tuning — outputs may be incorrect or inappropriate and should not be relied upon unchecked.

Hardware & framework

2× NVIDIA Tesla V100-PCIE-16GB · Windows · PyTorch DDP (gloo) · fp16 autocast (fp32 master) · fused AdamW · custom TinyBrainBot trainer.