dsc-370m-v4-5b-fineweb-edu

DSC v4 370M — FineWeb-Edu 5B pretrain

Continued pretrain of a custom DSC (Distributed Sparse Chunk-routing) variant built on GatedDeltaNet (GDN-2) 370M. This is the v4 architecture trained for 5B tokens on FineWeb-Edu.

Architecture (DSC v4)

• Base: GDN-2 370M (16 layers, d_model=1024)
• chunk_size=256, topk=2 routing
• alpha learnable init=0.5 (final avg=0.2656)
• desc_attn_v (descriptor attention value)
• ent_bias_scale=0 (entropy bias disabled)

Training

• Tokens: 5,000,000,000 (5B)
• Data: FineWeb-Edu sample
• Optimizer: AdamW lr=3e-4, weight_decay=0.01, cosine schedule
• Batch: 2/GPU x 8 = eff_batch 16, seq=4096
• Hardware: 8x H100 80GB (DDP)
• Time: ~9h (5B tokens @ ~155K tok/s)

Loading (custom architecture)

This is NOT a standard HF Transformers checkpoint. Load with the project's DSC code:

The DSC implementation is at: https://github.com/gyunggyung/LLM-OS-Models (under ).

Evaluation

After pretrain (before any fine-tune), RULER NIAH @ 4K:

The model has routing infra but doesn't use it effectively (decorative routing pattern). A2 fine-tune (NIAH memory-only aux loss) recovers single_1 to 0.65+ in 100M tokens. See .

Files

• — PyTorch state_dict (2.0GB, bf16)
• — alpha/router stats snapshot