nanochat-d34

nanochat-d34 model. It was pretrained like this:

torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- --depth=34 --device_batch_size=4 --target_param_data_ratio=40 --save_every=5000 --run=d34

On an 8XH100 node, which ran for ~100 hours (~4 days) and cost ~$2,500. Notably, note that it is 2X longtrained compared to Chinchilla, i.e. the param:token ratio is overridden from 20 up to 40. This means the model was trained longer that is compute optimal, just to squeeze a bit more capability into a bit smaller package.

Some of the notable stats of the model are as follows:

- depth: 34
- max_seq_len: 2048
- target_param_data_ratio: 40
- device_batch_size: 4
- total_batch_size: 524,288
- Number of parameters: 2,217,082,880
- Number of FLOPs per token: 1.426509e+10
- Calculated number of iterations: 169,150
- Number of training tokens: 88,683,315,200
- Tokens : Params ratio: 40.0000
- DDP world size: 8
- Minimum validation bpb: 0.7045
- Final validation bpb: 0.7045
- MFU %: 47.40%
- Total training flops: 1.265075e+21
- Peak memory usage: 69811.85MiB

The achieved CORE score of this base model is 0.3382. (For comparison, d32 is 0.3168, d20 "speedrun" is 0.22, GPT-2 is 0.25).

This upload allows you to skip base model pretraining and focus on finetuning, which is a lot cheaper, and saves you $2,500 of cost. To incorporate it into your nanochat repo it's a bit janky right now, basically download these files and then:

• the token_bytes.pt, tokenizer.pkl have to go into ~/.cache/nanochat/tokenizer directory
• the meta_169150.json and model_169150.pt have to go into ~/.cache/nanochat/chatsft_checkpoints/d34/

I'll figure out how to make this less janky in the future, and to make nanochat play nicer with huggingface infra.