ohara-chat-d12

ohara-chat-d12

A 124M-parameter chat model trained from scratch with ohara, replicating the nanochat pipeline end to end: pretraining on ClimbMix, then supervised finetuning into a chat model.

Trained on 2xA100-80GB in about 4.4 hours total.

Read the samples before you plan anything around this. At 124M effective parameters and 1.4e18 training FLOPs — roughly 3.5% of what nanochat spends to reach GPT-2 grade — this model has the form of knowledge without the content. It answers in the right shape, holds the conversation format, and stops cleanly. It also confabulates freely. It is a demonstration that the pipeline works, not a useful assistant.

Files

The pretrained base this was finetuned from lives separately, at joey00072/ohara-base-d12. Use that one if you want to run your own finetune.

Architecture

Llama-style decoder: RoPE, SwiGLU, RMSNorm, untied embeddings, nanochat-style initialization, trained with Muon (matrices) + AdamW (embeddings and scalars).

Training

Pretraining — 2,827 steps over 1.48B tokens of ClimbMix, batch 524,288 tokens, Muon at lr 0.02 with a warmup-stable-decay schedule.

For scale: nanochat reaches GPT-2 grade at 0.718 bpb, using ~28x more compute.

SFT — 800 steps on SmolTalk + MMLU + GSM8K (567,656 conversations), loss taken only on assistant tokens, conversations packed best-fit so none is split across rows.

Usage

from ohara.chat_engine import ChatEngine, SamplingConfig

engine = ChatEngine.from_checkpoint("sft_d12.pt", tokenizer_dir=".")
print(engine.generate(
    [{"role": "user", "content": "What is the capital of France?"}],
    SamplingConfig(temperature=0.7, max_new_tokens=100),
))

Or serve the browser chat UI:

python examples/chat_web.py --checkpoint sft_d12.pt --port 8080

Samples

Real, unedited output at temperature 0.7:

"What is the capital of France?"

The capital of France is Paris. Paris is a city located in the Latin Quarter, which is the largest city in France.

Correct, then immediately wrong.

"Why is the sky blue?"

It is named after the Swiss scientist Frank Herbert, who first described it in the 1930s.

Frank Herbert wrote Dune, was American, and was not a scientist.

"Write a haiku about rain." produces twelve lines, not three.

Conversation format

<bos> <|user_start|> ... <|user_end|> <|assistant_start|> ... <|assistant_end|>

Loss is taken only inside assistant turns, including the closing <|assistant_end|> so the model learns to stop.

Reproducing

DEPTH=12 bash runs/speedrun.sh