delphi-1e23-wc50m-warmup-levanter-sft

delphi-1e23-wc50m-warmup-levanter-sft

Math-weak supervised fine-tune (SFT) of the Delphi 1e23 (~25B-param) partial-K midtrain, trained with Levanter and exported to Hugging Face format.

This is the wc50m (wildchat) arm of the Delphi 1e23 post-training ladder: a general-chat SFT on a wildchat50m (90%) + delphi_warmup (10%) mixture, starting from the prepared 1e23 midtrained base. Contrast it with the math-strong laion/delphi-1e23-magpie-warmup-levanter-sft (magpie 90 / warmup 10).

Model

• Architecture: Qwen3ForCausalLM (dense), vocab 128256, 51 layers, hidden 5376, 42 attention / 42 KV heads, head_dim 128, max_position_embeddings 4096.
• Precision: fp32 (22 safetensors shards + index).
• RoPE: llama3 scaling — rope_theta = 500000, rope_scaling = {rope_type: llama3, factor: 8.0, low_freq_factor: 1.0, high_freq_factor: 4.0, original_max_position_embeddings: 8192} (mirrors the base; carried at both the top level and under rope_parameters for transformers-4/5 compatibility).
• Base: laion/delphi-1e23-p33m67-k0p076-prepared (prepared from the midtrained laion/delphi-1e23-p33m67-k0p076-lr0.67-b6607e).

Training

• Recipe: wc50m (math-weak). Mix: wildchat50m 90% / delphi_warmup 10%.
• Objective: completions-only SFT (assistant spans supervised).
• LR: 1e-5. Sequence length: 4096. Epochs: 1 (packed). Steps: 5306.
• Trainer: Levanter; exported to HF format.

Chat template — Delphi v0 (NOT ChatML)

This model uses the Delphi v0 chat protocol: Llama-3 headers (<|start_header_id|>/<|end_header_id|>/<|eot_id|>) plus repurposed single-id think/tool tokens:

⚠️ It is not ChatML — <|im_start|>/<|im_end|> are absent from this tokenizer. The Delphi v0 Jinja template is not baked into tokenizer_config.json; apply it externally (e.g. pass it to the serving engine). Reference implementation: experiments/sft/delphi_chat_template.py (the served template is that string with the Levanter {% generation %}/{% endgeneration %} markers removed). When decoding, keep skip_special_tokens=False so the atomic 128002/128003 think delimiters survive.

Provenance

Delphi 1e23 post-training study (issue #6279). HF-distribution only.