TR-HASH-MOE-500M-20B

TR-HASH-MOE-500M-20B

Research pretraining artifact for a 492.1M-parameter decoder-only language model with grouped-query attention, a shared dense MLP path, and deterministic token-ID hashed residual experts.

This is a base language model, not an instruction-tuned chat model. The repository provides both the final raw training checkpoint and a BF16 Safetensors export for direct base-model inference with the custom vllm-i64 runtime. For the instruction-tuned release, use TR-HASH-MOE-500M-HF.

Model summary

Each transformer block retains a shared dense MLP for contextual computation. The residual expert branch selects two experts from the token ID and layer index. No learned contextual router is evaluated at inference time. With four stored experts of width 128, each token activates two expert paths alongside the shared width-4,864 path.

Pretraining

• Corpus: FineWeb-Edu.
• Tokenizer: project 32k BPE tokenizer.
• Training tokens: 20B.
• Separate evaluation shard: approximately 40M tokens.
• Global tokens per optimization step: 262,144.
• Optimizer: AdamW.
• Precision: BF16.
• Run seed: 42.

The long run experienced host interruptions and was resumed from checkpoints. metrics.csv provides the merged trajectory, while metrics-raw-with-resumes.csv preserves the raw records around those resumes.

The plotted training curve is a 100-measurement rolling mean. Evaluation points are the recorded measurements on the disjoint evaluation shard. The chart starts after the initial warm-up to keep the later trajectory readable.

Recorded result

Step 76,293 is the final optimization checkpoint. The reported held-out value is from step 76,000, the last scheduled evaluation before completion; it is not a substituted training loss.

These numbers describe one exploratory run on one data distribution and one seed. They are not a claim of superiority over a dense or learned-router baseline at this scale.

Repository contents

The raw checkpoint contains model weights as well as optimizer, scheduler, configuration, argument, and backend state. It is intended for exact research resumption and framework-level inspection. Use model.safetensors with vllm-i64 for inference; this custom architecture is not a stock Transformers AutoModelForCausalLM implementation.

import torch

artifact = torch.load(
    "checkpoint.pt",
    map_location="cpu",
    mmap=True,
    weights_only=False,
)

print(artifact["step"])        # 76293
print(artifact["config"])
model_state = artifact["model"]

PyTorch checkpoints use pickle internally. Only load artifacts obtained from a source you trust.

Inference and downstream use

The base checkpoint has no instruction-following guarantee and should be prompted as a text-completion model. It intentionally contains no chat template, system prompt, or User/Assistant formatting. The public SFT export and live demo are:

• Instruction-tuned model
• TR-HASH-0.5B live demo
• Training framework
• Custom vllm-i64 runtime

Limitations

This is an experimental small base model. It can repeat text, lose coherence, invent facts, reproduce undesirable patterns from web data, or fail on reasoning, arithmetic, code, multilingual prompts, safety-sensitive requests, and long-context tasks. It has not been independently audited for memorization, contamination, bias, toxicity, privacy, or downstream safety.

Do not use it for consequential decisions. Evaluate the exact checkpoint and serving stack independently before any deployment.

License

The model artifact is released under CC BY-NC 4.0. FineWeb-Edu and individual source documents retain their own applicable terms; users remain responsible for reviewing dataset provenance and compliance for their intended use.