Museko-125M

Museko-125M

Museko-125M is a 123M parameter language model I trained from scratch. It targets the sub-150M size class, and across the standard small-model benchmarks it currently comes out ahead of the other models in that range.

It is a Llama style decoder pretrained on FineWeb-Edu with a small amount of synthetic arithmetic mixed in. The arithmetic mix is why it does noticeably better on math than most models its size while still holding solid general knowledge and reasoning.

Benchmarks

All numbers are 0-shot, measured with lm-evaluation-harness and the ArithMark 2.0 benchmark.

By the leaderboard averaging (ARC-Easy and ARC-Challenge combined) this comes to about 52.6, ahead of the previous top model at 49.71. It leads on four of the five benchmarks.

How to read these scores

The numbers above are multiple-choice scores, measured the way the Open SLM Leaderboard does it. The model is shown a prompt with a few candidate continuations and is scored on whether it ranks the correct one highest by likelihood. That is the standard setup and every model on the board is measured the same way, but it is not the same thing as free generation.

There is a real gap between the two. On ArithMark the model ranks the correct answer about 75 percent of the time, but when it has to generate the answer on its own it gets closer to 16 percent. That gap is normal for small models and for likelihood based benchmarks in general. So treat Museko as a strong base model for its size that completes text coherently and ranks well on these tasks, rather than one that reliably writes out correct answers by itself. It is best at completion, not at instructions or question answering.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

name = "TobiasLogic/Museko-125M"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16)

prompt = "The moon is"
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=120)
print(tok.decode(out[0], skip_special_tokens=True))

Generation notes

This is a base model, not an instruction or chat model, so use it for text completion rather than commands or question answering. Like most small base models it loops badly under plain greedy decoding, so the config here ships with a repetition penalty and an n-gram block by default and generation stays coherent out of the box. If you want more varied text, turn on sampling:

out = model.generate(**ids, max_new_tokens=120, do_sample=True, temperature=0.8, top_p=0.95, repetition_penalty=1.3)

Training setup

• Architecture: Llama, 12 layers, hidden size 768, 12 attention heads, 2048 context
• Parameters: about 123M
• Tokenizer: SmolLM2-135M (49,152 vocab)
• Data: FineWeb-Edu plus roughly 10% synthetic arithmetic
• Optimizer: AdamW, cosine schedule from 6e-4 down to 6e-5, 1,000 step warmup, weight decay 0.1
• Effective batch: about 0.52M tokens per step

Notes

Weights are open under Apache 2.0. This release is a from-scratch model and remains a strong general baseline for its size while being particularly good at arithmetic.