TR-HASH-MoE-100M-70B-Agentic-Refinement

TR-HASH MoE 100M - Agentic Refinement

Status: stopped and evaluated at optimizer update 9,791. The selected checkpoint is published at the repository root. The intended 70B-token schedule was stopped after 38.50B refinement tokens because the historical PIQA acc_norm gain remained limited to +0.16 percentage point over the pretraining final. This is an evaluated intermediate refinement checkpoint, not a completed 70B pass.

This run restarts from the verified final/ weights of TR-HASH MoE 100M - 125B Agentic Pretraining. It loads model weights only and creates a fresh optimizer and learning-rate schedule.

Root checkpoint

The repository root also contains the model configuration and its native 32K Agentic tokenizer. The token_pack_* directories remain available as resumable training checkpoints with optimizer state.

Evaluation

PIQA was evaluated over all 1,838 validation examples with the historical MLX scorer used for this lineage: goal + " " + solution.lstrip(), the model's own tokenizer, add_special_tokens=False, no BOS/EOS, and no chat template.

Machine-readable PIQA results and the exact scorer, tokenizer, and checkpoint hashes are provided in eval_results.json.

Corrected optimizer-update contract

The Complexity framework now prints an OPTIMIZER UPDATE CONTRACT block before data loading and rejects a 100M refinement launch if LR, scheduler, warmup, weight decay, or tokens per update differ from this contract. Runtime metrics retain the historical step fields and also expose optimizer_update, optimizer_updates_planned, and tokens_per_optimizer_update.

Provenance note

The earlier 100M refinement archive was withdrawn because it used 229,376 tokens per optimizer update, causing many more AdamW updates per source token. It must not be used as the initialization point for this run.

License

Released under CC BY-NC 4.0. Source datasets retain their own licenses and terms.