temporal-moe-ckpts

temporal-moe-ckpts

Megatron-LM training checkpoints for the Temporal-MoE experiments, 71 runs, 140.1 GiB. Checkpoints are in torch distributed checkpoint format (.distcp), not the older mp_rank_* layout.

---

WARNING, read before loading anything

1. Seventeen runs contain a second checkpoint that is not the trained model

Each of the 17 runs listed below contains two checkpoint directories: the genuine end-of-training checkpoint, and an iter_0000006 that is not it.

iter_0000006 was written by a later job that ran with finetune=True, which loads the trained weights and resets the iteration counter to zero. It therefore holds the fully trained model plus 6 additional optimizer steps, with the optimizer and learning-rate scheduler state discarded. It is not an obviously broken artifact. It will load, it will train, and it will emit plausible-looking evaluation numbers that are close to but not equal to the published ones. A run that loads it can produce results that look correct and are not, which is why this is dangerous rather than merely inconvenient.

The cumulative FLOP counter confirms the ordering in every affected run: the value stored in iter_0000006 is larger than the one in the fully trained checkpoint, because those 6 steps came after training finished.

latest_checkpointed_iteration.txt originally read 6 in all 17 runs and has been corrected in this repository, so --load now resolves to the trained checkpoint. See "Pointer corrections applied after upload" below for the per-run values and how they were verified. Both checkpoint directories remain present and unmodified, so passing --load with an explicit iteration is still the safest habit.

Unaffected runs have a single iter_* directory and a correct latest_checkpointed_iteration.txt.

2. g3_restorecheck/ckpt/iter_0000003/ is an incomplete write

This directory contains only common.pt. It is missing __0_0.distcp, __0_1.distcp, .metadata and metadata.json. It is an aborted checkpoint write and is not loadable.

It was uploaded deliberately rather than dropped, so the record of what exists on the training disk is complete. Do not attempt to restore from it. g3_restorecheck is an uncited infrastructure check and backs no published number.

---

Pointer corrections applied after upload

The ckpt/latest_checkpointed_iteration.txt files listed here were corrected in this repository after the checkpoints were uploaded. Each originally contained 6.

Reason: 6 resolved to iter_0000006, the finetune-mode continuation described in warning 1 above, written after training finished with the optimizer and learning-rate scheduler state discarded. A naive --load restored that instead of the trained model. Because it loads cleanly, trains, and produces plausible evaluation numbers, the failure was silent rather than visible.

Each corrected value was verified against four independent sources before publishing, and all four agreed in all 17 runs:

1. iters= in run.meta
2. the final successfully saved checkpoint from iteration N line in train.log
3. the iteration field inside that checkpoint's common.pt
4. the learning-rate schedule reaching min_lr (3.000059E-04 and 3.000005E-04 against a configured min_lr of 0.0003)

The corrected files match the format used by unaffected runs: bare digits, no trailing newline.

No checkpoint data was altered, and nothing was deleted. Every iter_0000006 directory remains uploaded and byte-identical to what was trained. Only the pointer files changed. Their sha256 and bytes entries in MANIFEST.csv describe the corrected content published here.

---

What this repository holds

71 run directories, plus _batch_logs/ and _lmeval_scratch/lmeval_out/. Each run directory holds the Megatron checkpoint plus the metadata needed to interpret it. 58 runs are cited by a results table or the paper, 13 are not. See the cited column described below.

Layout

<run_name>/
  ckpt/
    latest_checkpointed_iteration.txt   # see warning 1 above
    iter_<NNNNNNN>/
      __0_0.distcp                      # torch distributed checkpoint shards
      __0_1.distcp
      .metadata                         # DCP tensor index
      common.pt                         # iteration, arg namespace, RNG state
      metadata.json                     # backend versions
  run.meta                              # full knob set for the run
  train.log                             # Megatron stdout, includes the arg dump
  tb/                                   # TensorBoard event files
  lmeval_*/                             # lm-evaluation-harness outputs
  *.log, *.out                          # probe and evaluation logs
_batch_logs/                            # cross-run batch driver logs
_lmeval_scratch/lmeval_out/             # evaluation outputs from the scratch harness

Checkpoints copied into _lmeval_scratch for evaluation were byte-identical duplicates of the originals and were not uploaded. Only their evaluation outputs are here.

Training configuration

Every run directory contains run.meta and train.log. run.meta records the full knob set for that run in two lines, for example:

[run] temporal_fine_g3_1e19 N=185766400 iters=17112 warmup=856 min_lr=0.0003 eval@1711
[run] shape=s19opt H=800 L=14 ffn=4272 moe_ffn=184 grain=3 num_experts=192 shared_int=1100
      dense=0 temporal=1 shared_mult=2 topk=18 heads=50 gb=256 mb=8 lr=3e-3 flops=1e19

train.log is the full Megatron stdout, including the complete argument namespace dumped at startup. Read these before drawing any conclusion from a checkpoint.

MANIFEST.csv and the cited column

A manifest covering every file in all four repositories lives in the code repository. It has seven columns: local_path, hf_repo, hf_path, bytes, sha256, run_name, cited.

The cited column marks whether a run is referenced by a results table in results/ablations/*.csv or by the paper:

• cited, the run backs a published number. There are 58 of these.
• uncited, the run is infrastructure validation, a smoke test, a throughput probe, or an aborted run. It is kept for completeness and reproducibility, not because a table depends on it. There are 13 of these.
• empty, the file is not scoped to a single run, for example a batch log or an evaluation output.

Every sha256 in the manifest was computed on this disk before upload and each file was verified to exist remotely with a matching byte size.

Links

• Code: https://github.com/ncylich/temporal-moe
• Paper: Temporal-MoE (short paper), see the paper/ directory in the code repository
• Upstream platform this work forks: FLAME-MoE, arXiv:2505.20225

Companion repositories

• ncylich/temporal-moe-ckpts, Megatron training checkpoints
• ncylich/temporal-moe-router-adapt, router adaptation safetensors
• ncylich/temporal-moe-extras, captures, merged model, result tables, figures
• ncylich/temporal-moe-corpus, tokenized training corpus

Provenance

Trained with a personal fork of FLAME-MoE. Not affiliated with or endorsed by the FLAME-MoE authors.