tinybrainbot-303mV2-base

TinyBrainBot 303M V2 — Base

A ~303M parameter language model pretrained from scratch on a single 2×GPU workstation. This is the base (completion) model — no instruction tuning. For the chat variant, see nkthebass/tinybrainbot-303mV2-instruct.

Use this if you want a small English foundation model to continue text or fine-tune yourself.

Architecture

Llama-family (RoPE, RMSNorm, SwiGLU, GQA), tied embeddings.

Usage

transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-303mV2-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-303mV2-base")
ids = tok("Once upon a time, in a small village", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, top_p=0.9, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))

llama.cpp / Jan / LM Studio

GGUF files (F16, Q8_0) are included in this repo.

Training data

Pretrained (fp16, WSD schedule, DDP on 2× Tesla P100) on a mix of:

• FineWeb-Edu (10BT sample) — filtered educational web text
• Wikipedia (English)
• TinyStories v2 — simple synthetic stories (grammar/coherence)
• OpenWebText2
• Orca-Math — grade-school math word problems
• Synthetic distillation sets (generated by a larger teacher): short-answer facts + longer general Q&A

Note: because the pretraining mix included some chat-formatted synthetic Q&A, the base will occasionally continue a completion into <|user|>/<|assistant|>-style turns. That's expected for this base; the instruct model is the one tuned to actually chat.

Benchmarks

Multiple-choice log-likelihood scoring puts the family at roughly GPT-2-124M class on QA benchmarks (ARC-Easy ~46, ARC-Challenge ~27, beating GPT-2-124M on both), and at the random floor on HellaSwag/MMLU/WinoGrande — the size ceiling of a 303M. Full numbers are on the instruct card (multiple-choice scores barely move between base and instruct).

Limitations

• 303M parameters — it hallucinates and knows only common facts.
• English only, 1024-token context.
• No instruction tuning, no safety tuning. This is a raw base model.

License

Apache-2.0. Free to use and build on — attribution appreciated. Trained on public/open datasets (FineWeb-Edu, Wikipedia, TinyStories, OpenWebText2, Orca-Math, plus synthetic distillation data); please respect the licenses of those upstream sources.