bdh-100m-bpe8k-chat-v3

BDH 100M, BPE-8k, chat v3

bdh-100m-bpe8k tuned for 800 steps on the same 6,960 conversation-shaped documents as bdh-100m-bytes-chat-v3: story requests, stories followed by questions whose answers are only in the story, list requests followed by position questions, and a small set of one-line tasks and stop markers. Loss was taken only on the assistant's replies. The tuning data was written by an LLM from category prompts and is not in this release.

Format: turns are User: ... and Assistant: ... on their own lines. A reply ends with <|endoftext|>. chat_stream.py in the code repository handles this. Pass bpe8k.json as the second argument.

What tuning changed

• Recall of a fact planted earlier in the context (30 trials per distance): 93 % at 128 characters, 93 % at 256, 67 % at 384, 90 % at 600, and 33 % at 1,024, against 53 % at 128 for the base model. This is retrieval by key, not just naming the odd word in the context. With a second word planted under a different key (e.g. "The dog's name is Y.") in the same context, it answers with the word under the asked key 25 of 30 times at 64 characters and 24 of 30 at 600, against 4 of 30 for the other word.
• On a 60-item multi-turn exam (stories with three questions each, lists with position questions), 36 % of its story answers were judged supported by its own story and 64 % were correct when the story settled the question. It answered 64 % of "which was first", 43 % of "which was second", and 81 % of "which was last" questions correctly.
• Held-out bits per character rose from 0.964 to 1.172 (+22 %), so the tune makes it a worse general language model.

Training

800 steps of 32 sequences, AdamW with weight decay 0.01, learning rate 1e-4 flat, dropout 0.1, bf16, about four passes over the data. Takes about 51 minutes on an AMD Strix Halo integrated GPU.

Use

TEMP=0.3 python chat_stream.py bdh-100m-bpe8k-chat-v3.pt bpe8k.json

Limitations

A 100M-parameter model tuned on a few thousand short dialogues. It answers in the register of children's stories and invents facts. Its 480-token window (about 2,000 characters) holds a story and its three questions, which is why it holds up better across turns than the byte version. Single training seed for the base. The exam numbers are the mean of three generation seeds.

Code: https://github.com/d3vmeh/small-dragon-hatchling. Paper: https://doi.org/10.5281/zenodo.23325933 (CC BY 4.0).