deepseek-v4-mini-300M-cpt-fineweb

deepseek-v4-mini-300M-cpt-fineweb

Experimental / research artifact — not a usable model. Broad-data CPT of deepseek-v4-mini-300M-recovered on FineWeb-Edu, run locally on a single RTX 5090 laptop (2026-07-14).

What this is

The three-phase Lightning-Indexer protocol (see the recovery kit) applied with broad web text instead of the parent-family data — an experiment in whether the recovered slice generalizes off its native distribution at a ~98M-token budget.

Config: seq 1024, batch 4, grad-accum 8, Muon, WSD, bf16. W&B: m4mo8jd6

Findings (why this recipe is NOT recommended)

• Mechanically clean: phases A/B ran stably on broad data — useful evidence for scaling the protocol.
• Distribution tax: downstream SFT on the parent-family data (loggenix/DeepSeek-R1-style traces) restarted at loss 8.18 vs ~3.8 for the H100 run that skipped the broad-data detour. For this model family, train directly on parent-distribution data (loggenix_moe_sequelbox_sft_v1).
• Generations at loss ~7 are incoherent (expected at this budget).

Lineage

DeepSeek-V4-Flash → 300M slice (-from-flash) → indexer recovery (-recovered) → this → sft-v1 → sft-v2