tinybrainbot-320mV2-math

TinyBrainBot 320M V2 — Math

A ~326M-parameter decoder-only model, trained from scratch on ~10B tokens (2× Tesla V100), then fine-tuned to be a math-reasoning model: multi-digit arithmetic and grade-school word problems, solved by showing the work (column arithmetic, long division, partial-product multiplication) rather than guessing.

• Base model: tinybrainbot-320mV2-base.
• fp16 safetensors (AutoModelForCausalLM) and F16 GGUF (LM Studio / Ollama / llama.cpp) both provided.

TL;DR: For its size it does arithmetic and structured word problems far above its weight — it beats GPT-3-175B on 3–5-digit arithmetic (both tool-free) and solves multi-step word problems with commas and mixed operations. It is not a general-knowledge model — treat it as a compact math engine that also chats a little.

---

What it does well

It reads the problem and computes — e.g. "A store had 56,321 items and sold 28,479. How many remain?" →

<think> Start with 56321. Then subtract 28479. Subtract column by column:
ones: 11 - 9 = 2, borrow 1. ... So 56321 - 28479 = 27842. </think>
The answer is 27842.

Evaluation

GPT-3 Arithmetic protocol (exact-match) — vs GPT-3-175B (few-shot, direct):

Ours uses trained-in worked steps; GPT-3's numbers are direct-answer. Both are pure LMs with no external tools/calculators. The point is about method: teaching a 326M model the algorithm beats a 175B model guessing — decisively on 4–5-digit arithmetic.

• Word-problem set (large-number add/sub with commas, multi-step, 2-digit multiply, first-person phrasings): solves essentially all of a 20-problem targeted set by reading the problem and computing the steps.
• GSM8K: ~3–4% (zero-shot CoT, n=500) — off the base instruct's 0.53% floor, at roughly the SmolLM2-360M-Instruct tier. Arbitrary hard multi-step word problems remain scale-limited at 326M.

General benchmarks (log-likelihood MC, our harness; the math SFT did not erode general ability):

Reaches the Pythia-410M tier — a model trained on ~30× more tokens — while being math-specialized.

Usage

Chat format:

<|user|>
{question}
<|end|>
<|assistant|>
{answer}
<|end|>

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-320mV2-math")
m = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-320mV2-math", torch_dtype=torch.float16)
ids = tok.apply_chat_template([{"role":"user","content":"A theater has 56 rows with 27 seats in each row. How many seats?"}],
                              add_generation_prompt=True, return_tensors="pt")
print(tok.decode(m.generate(ids, max_new_tokens=256, do_sample=False)[0][ids.shape[1]:], skip_special_tokens=True))

GGUF file (*-F16.gguf) works directly in LM Studio / Ollama / llama.cpp — use the model's built-in chat template as-is.

GGUF tokenization fix (this release): the F16 GGUF now sets tokenizer.ggml.add_space_prefix=false and ships a leading-space chat template, so llama.cpp tokenizes the chat format token-for-token identically to the native SentencePiece tokenizer. This fixes a prior export mismatch (llama.cpp #23840: the default add_space_prefix=true injects phantom ▁ around special tokens) that garbled arithmetic in GGUF apps. Multi-digit add/subtract now compute correctly in-app (e.g. 56321 − 28479 → 27842). Note: multiplication is the model's fp16-precision soft spot — it's stronger in fp32 than at the fp16 the GGUF runs — so hard multiplies can still miss.

Prompting tips

This is a math model — strongest on multi-digit arithmetic and worked-step word problems; general-knowledge chat is weak. Ask direct math questions (e.g. what is 19 × 82) for best results.

Model details

Training process

• Pretraining — from scratch, 51,000 steps / ~10.03B tokens on 2× Tesla V100 (PyTorch DDP gloo, fp16 + GradScaler, fused AdamW). Warmup–Stable–Decay schedule: 1,000-step warmup → stable LR 6e-4 → cosine decay over the final ~20% (from step 40,800). A quality-anneal (swap to a knowledge-dense data mix) runs over the last ~3B tokens — the visible dip near step 40k. 13-source data mix, principle real > synthetic (≤ ~35%): DCLM web, Wikipedia leads, FineWeb-edu, filtered Python/JS code, verified arithmetic, and distilled Q&A/facts/reasoning. Pretrain loss ~10.6 → ~2.3.
• Math-reasoning SFT (steps 51k → 58k, green) — supervised fine-tuning (assistant-masked, chat format) that teaches: multi-digit arithmetic as explicit worked steps (column add/sub, long division, partial-product multiply); word problems that read the problem then delegate the arithmetic to column computation (large numbers, commas, multi-step, first-person phrasings); plus retained general chat / greetings. The data was iteratively refined to kill template-overfit (phantom steps), cover diverse verbs and first-person forms, and handle large/comma-formatted numbers. SFT loss → ~0.4.

Limitations

• General knowledge is weak — it can drift into confident errors on factual/open-ended questions. This is a fundamental 326M capacity limit, not a bug. Use it for math, not facts.
• Novel word-problem phrasings can still trip it (it may drop a step on unusual structures).
• Hard multi-step reasoning (GSM8K/MATH) caps at this scale.
• 3+ digit multiplication and large-number division are soft spots.
• English only, 1024-token context, no RLHF/safety tuning — outputs may be wrong or inappropriate; don't rely on them unchecked.

Hardware & framework

2× NVIDIA Tesla V100-PCIE-16GB · Windows · PyTorch DDP (gloo) · fp16 · custom TinyBrainBot trainer.