seedvar-moe-1e18-d832-seed45

seedvar-moe-1e18-d832-seed45

Seed-variance run for Sparse Layers are Critical to Scaling Looped Language Models (arXiv:2605.09165), trained to measure run-to-run noise in the 1e18-FLOP benchmark numbers.

Important: what varies across these four seeds

Only the training data order. The initialization seed is fixed at 42 for all four runs, as is the validation-batch order. The spread across seeds 42-45 therefore measures data-order variance, which is a lower bound on full run-to-run variance — a study that also varied initialization would be expected to show equal or greater spread. Do not read these error bars as total training noise.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
    "ml-ryanlee/seedvar-moe-1e18-d832-seed45", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("gpt2")

Evaluated with OLMES 5-shot core_9mcqa::olmes. When evaluating, pass max_length=1024 — the RoPE buffer is sized to the 1024-token training context and longer sequences overflow it.