gdn2-370m-fineweb-edu-5b

GDN-2 370M (FineWeb-Edu 5B) — vanilla baseline

---

1. Purpose

Vanilla GDN-2 370M baseline trained on the same 5B-token slice as dsc-370m-fineweb-edu-5b-v3, so the DSC vs vanilla comparison is apples-to-apples (same data, same compute, same hyperparameters).

2. RULER results (vs DSC v3 5B)

DSC v3 vs Vanilla 5B comparison is the 5B verdict that determines whether DSC is a useful architectural addition to GDN-2. Full results in docs/DSC_LIVE_STATUS_KO.md. This vanilla checkpoint is the floor that DSC improvement is measured against.

3. Known limitations

• 5B tokens below Chinchilla optimal (~7.5B for 370M params)
• Long-context (≥16K) retrieval fails completely (vanilla linear attention state collapse) — the gap DSC is designed to close

4. Citation

@misc{gdn2-vanilla-5b-2026,
  author = {LLM-OS-Models},
  title  = {GDN-2 370M FineWeb-Edu 5B vanilla baseline},
  year   = {2026},
  url    = {https://huggingface.co/LLM-OS-Models/gdn2-370m-fineweb-edu-5b}
}