bdh-100m-bpe8k

BDH 100M, BPE-8k

A 109.1M-parameter Dragon Hatchling (BDH) language model over an 8,192-token byte-pair vocabulary (bpe8k.json, trained on the same corpus with the Hugging Face tokenizers library). Base model, no tuning. The byte-level twin trained on the same recipe is bdh-100m-bytes. The two differ only in the alphabet and the vocabulary matrices: 100.9M shared-layer parameters each, with 8.4M of vocabulary parameters here against 0.3M for the byte model.

Architecture: the reference BDH from pathwaycom/bdh (arXiv:2509.26507). Six layers sharing one set of weights, four heads, residual width 512, 16,384 sparsely activated neurons per head, 512-position training block.

Training

• Text: 1.41B tokens (about 5.4 GB), 83 % FineWeb-Edu and 17 % TinyStories (train split), the same amount and mix as the byte model but not the identical documents. 512-token blocks with no state carried between blocks.
• 86,000 steps of 32 sequences (16,384 tokens per step), AdamW with weight decay 0.1, learning rate 1e-3 to 1e-4 on a cosine schedule with 1,500 warmup steps, dropout 0.1, bf16 autocast.
• One RTX 4090, 19.8 hours. One training seed.

Results

Bits per character on 3 MB of held-out text neither model saw in training (part of FineWeb-Edu shard 013 and the TinyStories validation split), scored in 512-position blocks:

The byte twin scores 1.042 combined. When both models are given the same number of preceding characters, they score within 0.02 bpc of each other. The gap is likely due to each token position seeing about four times more text. A matched rotary transformer on the same text scores 0.988 (2.4 % higher than this model).

The model stores a fact planted earlier in its context and, untuned, recalls it more often than the byte model: top-1 recall of the planted word is 57 % at 64 characters, 53 % at 128, and 33 % at 256, from 30 trials each. Density of active neurons per position is 11.9 %.

Use

import torch, bdh                         # bdh.py from the code repository
from tokenizers import Tokenizer
tok = Tokenizer.from_file("bpe8k.json")
ck = torch.load("bdh-100m-bpe8k.pt", map_location="cpu")
model = bdh.BDH(bdh.BDHConfig(**ck["config"]))
model.load_state_dict(ck["model"])

For generation, use chat_stream.py from the code repository with RAW=1 and the tokenizer path as the second argument.

Limitations

Small model, 5.4 GB of text. The vocabulary was built from the training corpus, which may favor it slightly on text of the same kind. Only one vocabulary size was tried. Single training seed.

Code: https://github.com/d3vmeh/small-dragon-hatchling. Paper: https://doi.org/10.5281/zenodo.23325933 (CC BY 4.0).