TinyBrainBot-demo-216.5m

TinyBrainBot — demo (216.5M)

A 216.5M-parameter conversational language model trained from scratch on a custom decoder-only architecture. This is an early proof-of-concept / demo checkpoint: the goal is to show that a small model with a proper subword tokenizer can hold coherent, on-topic chat — not to compete with production models.

Trained end-to-end (tokenizer + architecture + pretraining + SFT) as a solo project on consumer hardware.

Model details

Note: ~551M training tokens is well under compute-optimal for this size, so this is an early, undertrained checkpoint. Expect coherent short replies but factual drift and looping on long generations. More training is planned.

How to run

LM Studio

Load the GGUF, then set sampling to:

The chat template is embedded in the GGUF, so no manual prompt-format setup is needed.

llama.cpp

llama-cli -m tinybrainbot-216.5m-F16.gguf --jinja \
  --temp 0.70 --top-p 0.90 --repeat-penalty 1.15 \
  --frequency-penalty 0.5 --presence-penalty 0.3

Chat format

The model uses these special tokens (single IDs in the tokenizer):

<|user|>
{your message}
<|end|>
<|assistant|>

Generation stops at <|end|>.

The GGUF tokenizer is exported as UGM (unigram) so llama.cpp reproduces the training-time SentencePiece tokenization exactly — an SPM export would re-segment words and degrade output.

Intended use

• Demonstrations and education about small-model training/inference
• Research into tokenizers, architectures, and from-scratch training on modest hardware
• A base for further training / fine-tuning experiments

Not intended for factual question answering, production use, or any high-stakes application.

Limitations

• Small + undertrained: answers are often factually wrong and can wander off-topic on long outputs.
• Short context (768 tokens).
• English-only.
• Greets and opens coherently; degrades the longer it generates (mitigate with the frequency/presence penalties above).

Training data

Pretraining on public English text (e.g. Wikipedia, TinyStories, OpenWebText2) and SFT on public chat/instruction datasets. Respect the upstream licenses of those datasets for your own use.