linnet-497M

linnet-497M

A 497M parameter Mixture of Experts base language model with 8 experts and 2 active experts per token and 157M active parameters. Trained from scratch using rudyon/pipeline on the HuggingFaceFW/fineweb-edu and mlfoundations/dclm-baseline-1.0 datasets.

Training was done on a single H100 GPU rented on Prime Intellect for about $17.

training status

⚠️ This model is undertrained. Chinchilla-optimal training would require ~19000 steps on ~10B tokens. This checkpoint was saved at step ~5000 (~26% of optimal), due to compute budget constraints. The loss curve was still descending at the time of stopping.

architecture

The model is a 12-layer causal transformer with the following architecture:

training

• Datasets: HuggingFaceFW/fineweb-edu (~700k docs) + mlfoundations/dclm-baseline-1.0 (~250k docs)
• Tokenizer: Custom ByteLevelBPE (vocab size: 32768)
• Batch size: 524,288 tokens
• Sequence length: 1024

usage

Download model.py from the repository alongside the weights, then:

import torch
from tokenizers import Tokenizer
from model import LLM, LLMConfig

device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = Tokenizer.from_pretrained("rudyon/linnet-497M")
model = LLM(LLMConfig(depth=12, vocab_size=32768))
state_dict = torch.load("pytorch_model.bin", map_location=device)
model.load_state_dict(state_dict)
model.eval()
print(model.generate("Hello!", enc=tokenizer))