Downloads · 30 days
6
1% of all-time downloads
sfanm/d12-pretrain-climbmix15B-dolmino5B
d12-pretrain-climbmix15B-dolmino5B is a text generation model from sfanm. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
The loadable model at this repository's root is the terminal checkpoint of a 124M-parameter decoder trained from scratch on a 20B-token mixture: 75% ClimbMix web-scale pretraining text and 25% of the OLMo-3 Dolmino mi…
Downloads · 30 days
6
1% of all-time downloads
All-time downloads
691
Public
Parameters
124M
2 GB on disk
Likes
0
Public
Click a slice to open those files.
.distcp1.7 GB · 87%
From the Hugging Face model README
The loadable model at this repository's root is the terminal checkpoint of a 124M-parameter decoder trained from scratch on a 20B-token mixture: 75% ClimbMix web-scale pretraining text and 25% of the OLMo-3 Dolmino mid-training mix, interleaved uniformly through the whole run. The run consumed exactly 20,000,014,336 training tokens (38,147 optimizer iterations at 256 × 2,048), of which 15.0B came from ClimbMix and 5.0B from Dolmino.
| Field | Value |
|---|---|
| Parameters | 123,587,328 |
| Layers / hidden size | 12 / 768 |
| Attention heads | 12 (MHA, head dim 64) |
| FFN size | 2,048 (SwiGLU/SiLU) |
| Position encoding | RoPE, theta 10,000 |
| Normalization | RMSNorm |
| Embeddings | Tied |
| Tokenizer | GPT-2 BPE, vocabulary padded to 50,304 |
| Context | 2,048 tokens |
| Published weights | BF16 |
The Transformers class is LlamaForCausalLM, but this is a from-scratch model
with the GPT-2 tokenizer (the same family as the sfanm/d24-* models, at
half the depth and hidden size), not a Llama-family pretrained checkpoint.
Both corpora were tokenized with the GPT-2 tokenizer and sampled at the token
level, not physically resized: every training batch is about 75% ClimbMix and
25% Dolmino. Every document ends with exactly one <|endoftext|> token
(id 50256, the model's eos_token_id); no beginning-of-sequence token was
prepended, which matches the tokenizer's default behaviour. Token ids
50257–50303 exist only as vocabulary padding and never occur in the data.
karpathy/climbmix-400b-shuffle,
62 training shards; 15.0B tokens were drawn, at most 15.3% of any shard.allenai/dolma3_dolmino_mix-100B-1025 (OLMo-3's mid-training mix
at its released component proportions), 62 training shards; 5.0B tokens were
drawn, at most 4.5% of any shard.No document was seen twice. One held-out shard per corpus served as the validation and test split, mixed with the same 75/25 weights.
Sequence length 2,048, global batch 256, AdamW (β 0.9/0.95, ε 1e-8) at peak
LR 6e-4, weight decay 0.1, gradient clip 1.0. The schedule is WSD: 200
warmup steps, constant 6e-4 through iteration 30,518, then a 7,629-step
cosine cooldown to 6e-5. Final held-out language-model loss on the mixed
validation split was 2.5535 (perplexity 12.9); on the test split 2.5690
(perplexity 13.1). Training loss fell from 2.73 at the end of the constant
phase to 2.64 at the end of the cooldown.
The run was trained with Megatron-Bridge (NeMo 26.04 container) on ALCF Polaris, on 8 nodes × 4 A100-40GB, in about 36 node-hours.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "sfanm/d12-pretrain-climbmix15B-dolmino5B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, dtype=torch.bfloat16, device_map="auto"
)
This is a base language model and has no chat template. The resumable
Megatron distributed checkpoint of the terminal iteration is published under
megatron/iter_0038147 (model, optimizer and scheduler state).
This experimental research checkpoint has not undergone safety alignment or a comprehensive evaluation. Review the licenses and terms of all upstream data before redistribution or downstream use.