ohara-base-d12

ohara-base-d12

The pretrained base model from a nanochat-style run with ohara. 124M effective parameters, trained from scratch on 1.48B tokens of ClimbMix on 2xA100-80GB in about 3.3 hours.

This is a base language model, not a chat model. It continues text; it does not answer questions and it never stops on its own. Given "Why is the sky blue?" it will happily continue into an unrelated encyclopedia article. If you want something that responds to you, use ohara-chat-d12, which is this checkpoint after supervised finetuning.

Use this one if you want to run your own finetuning from a base.

Architecture

Llama-style decoder: RoPE, SwiGLU, RMSNorm, untied embeddings, nanochat-style initialization, trained with Muon (matrices) + AdamW (embeddings and scalars).

The 8 conversation special tokens are reserved in the vocabulary but never seen during pretraining, so a finetune can use them without resizing the embedding.

Training

2,827 steps over 1.48B tokens, batch 524,288 tokens, Muon at lr 0.02 under a warmup-stable-decay schedule, bf16 mixed precision.

Final val loss 2.9153, next-token accuracy 43.6%.

For scale: nanochat reaches GPT-2 grade at 0.718 bpb using roughly 28x more compute than the 1.4e18 FLOPs spent here. This is an honestly small model.

Usage

import torch
from ohara.chat_engine import config_from_state_dict
from ohara.models.llama import Llama

state = torch.load("base_d12.pt", map_location="cpu")["model"]
model = Llama(config_from_state_dict(state))
model.load_state_dict(state)

To finetune it into a chat model:

python examples/train_sft.py --pretrained-checkpoint base_d12.pt

Reproducing

DEPTH=12 bash runs/speedrun.sh