bdh-50m-bpe8k

BDH 50M, BPE-8k

A 62.9M-parameter Dragon Hatchling (BDH) language model over an 8,192-token byte-pair vocabulary (bpe8k.json, trained on the same corpus with the Hugging Face tokenizers library). Base model, no tuning. The byte-level twin trained on the same recipe and on exactly the same text is bdh-50m-bytes. The two differ only in the alphabet and the vocabulary matrices: 56.8M shared-layer parameters each, with 6.3M of vocabulary parameters here against 0.2M for the byte model.

Architecture: the reference BDH from pathwaycom/bdh (arXiv:2509.26507). Six layers sharing one set of weights, four heads, residual width 384, 12,288 sparsely activated neurons per head, 512-position training block.

Training

• Text: 3.07 GB (about 0.80B tokens), 83 % FineWeb-Edu and 17 % TinyStories (train split), 512-token blocks with no state carried between blocks. 54 bytes of text per shared-layer parameter.
• 48,725 steps of 32 sequences (16,384 tokens per step), AdamW with weight decay 0.1, learning rate 1e-3 to 1e-4 on a cosine schedule with 1,500 warmup steps, dropout 0.1, bf16 autocast.
• One RTX 4090, 7.4 hours. One training seed. A second seed of the same recipe scored 1.019 combined.

Results

Bits per character on 3 MB of held-out text neither of the models saw in training (part of FineWeb-Edu shard 013 and the TinyStories validation split), scored in 512-position blocks:

The byte twin scores 1.104 combined. When both models are given the same number of preceding characters, they score within 0.02 bpc of each other. The gap is likely due to each token position seeing about four times more text. A matched rotary transformer on the same text scores 1.029 (1.0 % higher than this model).

The model stores a fact planted earlier in its context (a planted word becomes thousands of times more likely to be predicted) but, untuned, rarely says it back: top-1 recall of the planted word is 6 of 30 at 64 characters and 2 of 30 at 128. Density of active neurons per position is 12.3 %. It is about 50 % for an untrained model of the same shape.

Use

import torch, bdh                         # bdh.py from the code repository
from tokenizers import Tokenizer
tok = Tokenizer.from_file("bpe8k.json")
ck = torch.load("bdh-50m-bpe8k.pt", map_location="cpu")
model = bdh.BDH(bdh.BDHConfig(**ck["config"]))
model.load_state_dict(ck["model"])

For generation, use chat_stream.py from the code repository with RAW=1 and the tokenizer path as the second argument.

Limitations

Small model, 3.07 GB of text. The vocabulary was built from the training corpus, which may favor it slightly on text of the same kind. Only one vocabulary size was tried. Single training seed for this checkpoint.

Code: https://github.com/d3vmeh/small-dragon-hatchling. Paper: https://doi.org/10.5281/zenodo.23325933 (CC BY 4.0).