tinybrainbot-350mV3-instruct

TinyBrainBot 350M V3 — Instruct

A 348M-parameter decoder-only model, pretrained from scratch on 22.7B tokens and then instruction-tuned. Answers direct questions in a chat format. Successor to the TinyBrainBot 100M V3 Instruct.

TL;DR: Beats the 100M V3 Instruct on 4 of 7 benchmarks and on average (41.8 vs 41.3), with its biggest gains on HellaSwag (+3.1) and OpenBookQA (+2.4). It answers short factual questions well and explains concepts at a reasonable level for its size. For anything numeric, use the math variant.

---

Model details

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

m = "nkthebass/tinybrainbot-350mV3-instruct"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m)

msgs = [{"role": "user", "content": "What is the capital of France?"}]
p = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(p, return_tensors="pt"), max_new_tokens=150,
                     do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

Use apply_chat_template — do not hand-build the prompt. This family's template is space-separated, not newline-separated:

 <|user|> What is the capital of France? <|end|> <|assistant|>

(note the leading space). Building it with newlines instead degrades output badly — the model emits fragments or repeated tokens. The same applies to the GGUF, which carries this template along with add_space_prefix=false; both halves are required.

Evaluation

EleutherAI lm-eval-harness, 0-shot, complete test sets — same harness as the published 100M V3 numbers. Compared instruct-to-instruct, not against a base model.

→ 4/7 vs the 100M V3 Instruct.

Note on instruct vs base scores. Instruction tuning lowers multiple-choice benchmark scores for this family — the 350M V3 base averages 43.2 and scores 56.6 on ARC-Easy, versus 41.8 and 50.9 here. That is the normal cost of making a model usable, and it is why the base and instruct cards report separate tables. Do not compare this model against a base model's numbers.

Training data

12 sources, 131M tokens, 2,000 steps at lr 2e-5 -> 0 on a WSD schedule.

smoltalk, magpie-pro-300k and openhermes-clean are public instruction sets. The remainder are generated locally: qa-distill and reasoning-distill-chat are distilled question-answer and reasoning exchanges, longdef-sft is long-form definitional answers, convo-recall and convo-core are multi-turn conversational data, and greetings is a small set of short social exchanges.

What is deliberately small here. Math and reasoning together are ~11%, enough that the model attempts arithmetic without the instruction data being dominated by it. For anything numeric, use the math variant - this mix does not make a calculator.

Sample exchanges

All generated from this repo with temperature=0.7, top_p=0.9, unedited.

USER: What is the capital of France?
BOT : Paris is the capital of France.

USER: What causes tides?
BOT : Tides are caused by the gravitational attraction between the Earth and the Moon.
      The Moon orbits the Earth, and when it is closest to Earth, its gravitational pull
      is strongest, resulting in a higher high tide, while when it is farthest away, its
      pull is weaker, resulting in a lower high tide...

USER: Give me three tips for studying effectively.
BOT : Here are three effective study tips:
      1. **Set clear goals**: Identify what you want to achieve, such as improving your
         grades or mastering a particular subject.
      2. **Take regular breaks**: Regularly take breaks from your studying schedule to
         avoid burnout and maintain your focus.
      3. **Use active learning techniques**: ...

USER: Can you explain photosynthesis simply?
BOT : Photosynthesis is a process that converts light energy into chemical energy, storing
      energy in the form of ATP and NADPH. In photosynthesis, light energy is used to
      combine carbon dioxide and water into glucose and oxygen. This process occurs in
      three stages: light absorption, energy conversion, and carbon fixation...

And where it falls down — the same run, same settings:

USER: Who was Marie Curie?
BOT : Marie Curie was a French mathematician and physicist who made significant
      contributions to physics, mathematics, and medicine.
      (Polish-French, and a chemist rather than a mathematician)

USER: What's the difference between a virus and a bacterium?
BOT : ...Viruses are composed of genetic material, such as DNA, and contain genetic
      material, called genetic material, which can be replicated...
      (correct direction, then degenerates into tautology)

USER: What is the Great Barrier Reef?
BOT : A remarkable underwater landmark that spans over 4,000 kilometers in length and
      2,000 kilometers in width.
      (roughly 2,300 km long; the width is wildly wrong)

The pattern is consistent: the shape of the answer is usually right and the specific details often are not. It reaches for the correct concept — gravity for tides, chlorophyll for autumn leaves, ATP/NADPH for photosynthesis — and then invents numbers, nationalities and qualifiers. Treat it as a model that knows the outline of things at 348M, not a reference.

Limitations

• Not a calculator. It will attempt arithmetic and often gets small sums right, but it does not reliably stop once it has the answer — it keeps generating and can talk itself into a wrong one. Use the math variant for anything numeric.
• Weak on open-ended conversation. It answers direct questions; it does not sustain multi-turn chat well, tends to restate the user's statement, and loses the thread across topic shifts.
• Hallucinates confidently on facts outside its training distribution.
• MMLU and ARC-Easy lag the 100M V3 Instruct.
• Trained almost entirely on English.

Related

• TinyBrainBot 350M V3 Base — the pretrained model, and the frozen-RMSNorm story behind it.
• TinyBrainBot 350M V3 Math — for arithmetic and worked solutions.