Downloads · 30 days
5
8% of all-time downloads
sfanm/d24-pretrain-v1-climbmix-5.8B
d24-pretrain-v1-climbmix-5.8B is a text generation model from sfanm. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
v1 from-scratch pretrained BASE LM (undertrained).
Downloads · 30 days
5
8% of all-time downloads
All-time downloads
59
Public
Parameters
757M
3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.5 GB · 100%
From the Hugging Face model README
v1 from-scratch pretrained BASE LM (undertrained).
nanochat-style depth-24 decoder — 24 layers × 1536 hidden × 12 heads, SwiGLU / RoPE / RMSNorm, tied embeddings, GPT-2 BPE vocab (50304), 0.757B params, 2048-token context.
Lineage. From scratch on 5.84B ClimbMix tokens (karpathy/climbmix-400b-shuffle), GPT-2 vocab.
Metrics. val loss 2.73 (~7.8 tok/param → undertrained). Base for the v1 midtrain/SFT lineage.
This is a base language model (pre-SFT) — use it for text continuation, not chat. EOS is the GPT-2 <|endoftext|> (50256). For a chat model, use the d24-sft-* checkpoints.
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "sfanm/d24-pretrain-v1-climbmix-5.8B"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype="bfloat16", device_map="auto")
inputs = tok("The derivative of x**2 is", return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**inputs, max_new_tokens=128)[0], skip_special_tokens=True))
Research checkpoint from a from-scratch nanochat-d24 replication (pretrain → midtrain → SFT → RL) on NERSC Perlmutter. Trained on third-party corpora (ClimbMix, FineMath, OpenMath, MetaMath, OpenThoughts, OLMo-3 Dolmino, SmolTalk, …) — see those datasets' licenses; provided as-is for research.