TR-HASH-MoE-200M-160B-SFT

TR-HASH MoE 200M — 160B-source Full SFT (32K release)

Paper

Deterministic multi-hash routing supports long-horizon training in a compact language model

Preprint (DOI) · Model weights · Code

Full-parameter instruction SFT of AETHORIA-AI/TR-HASH-MoE-200M-160B-Refinement on the audited AETHORIA-AI/TR-HASH-MoE-200M-SFT-v2-300K. This release is not LoRA or QLoRA: all 201.2M model parameters were trainable.

The root model.safetensors is the epoch 3 / step 5,982 checkpoint, selected for the strongest full-PIQA result and lowest held-out SFT loss across the three epochs.

Tokenizer: use AETHORIA-AI/TR-HASH-Tokenizer-32K with this 32,000-token model.

Results

Released checkpoint benchmark panel

All benchmark evaluations use zero-shot causal continuation log-likelihood, no chat template and a maximum sequence length of 2,048. Accuracy selects the choice with the highest total continuation log-likelihood; the normalized metric selects by mean continuation log-likelihood per scored token. PIQA was evaluated from the native epoch-3 checkpoint. ARC and HellaSwag were evaluated from an FP16 MLX conversion of the same root F32 SafeTensors weights. ARC Combined is the micro-average over all 3,548 ARC-Easy and ARC-Challenge test examples, not the arithmetic mean of the two percentages. Machine-readable reports are published under reports/sft-v2-300k/.

Compact-model comparison

Combined ARC is weighted by the public test-set sizes (2,376 ARC-Easy and 1,172 ARC-Challenge examples). GPT-2 Small and OPT reference scores come from the AMD-LLM lm-evaluation-harness comparison; their combined values are approximate because the published component scores are rounded. Pythia-160M uses EleutherAI's official zero-shot result. GPT-2 Medium, GPT-2 Large and Pythia-410M were evaluated locally in MLX FP16 with the same causal-choice evaluator as TR-HASH. OPT-350M used the identical prompt and scoring formula in PyTorch MPS FP16 because MLX does not implement the OPT architecture. The 124M–160M references were reported through lm-evaluation-harness, so that part of the comparison is informative rather than a claim of bit-identical evaluation runtimes.

Training recipe

Architecture and loading

201.2M parameters, 16 decoder layers, GQA (14 query heads / 2 KV heads), four stored deterministic token-ID-routed experts with top-2 activation, an always-on shared SwiGLU path and tied embeddings. The persisted multi-hash routing tables are part of the checkpoint.

The repository includes an autonomous Transformers adapter. Load it with:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "AETHORIA-AI/TR-HASH-MoE-200M-160B-SFT"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

License

The released model weights and repository code are licensed under CC BY-NC 4.0 as reproduced in LICENSE. Commercial use is not permitted under this license. Source datasets retain their own licenses and terms; see the dataset manifest for the per-source audit.

Citation

@article{Peyriguere_2026,
  title     = {Deterministic multi-hash routing supports long-horizon training in a compact language model},
  author    = {Peyriguere, Boris},
  year      = {2026},
  month     = aug,
  publisher = {Springer Science and Business Media LLC},
  doi       = {10.21203/rs.3.rs-10788774/v1},
  url       = {https://doi.org/10.21203/rs.3.rs-10788774/v1}
}