d24-v6-midtrain

D24 v6 — midtrain

The loadable model at this repository's root is the terminal replay-free midtraining checkpoint in the public D24 v6 lineage. Starting from sfanm/d24-v6-pretrain, it consumed 100,000,595,968 OLMo-3/Dolmino midtraining tokens with an explicit pretraining replay fraction of zero and ended at optimizer iteration 23,842.

Architecture

The Transformers class is LlamaForCausalLM, but this model uses the D24 architecture and GPT-2 tokenizer.

Training

Midtraining used sequence length 2,048, global batch 2,048, microbatch 4, AdamW at peak LR 4e-4, weight decay 0.1, gradient clip 1.0, and a WSD schedule with 1,000 warmup steps and a final 4,768-step cosine decay to 4e-5. Final held-out language-model loss was 1.365992 (perplexity 3.920).

Loading

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sfanm/d24-v6-midtrain"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

This remains a base language model and has no chat template. All 12 retained, resumable Megatron distributed checkpoints are published under megatron/, from iter_0002000 through the terminal iter_0023842.

This experimental research checkpoint has not undergone safety alignment or a comprehensive evaluation. Review the licenses and terms of all upstream data before redistribution or downstream use.