serblo-125m

Serblo-125M

This is the first open Bosnian-Croatian-Serbian model where the entire stack is from scratch and public: corpus pipeline, tokenizer, architecture, training loop, SFT and preference datasets. All 70.8B training tokens are BCS. Open Serbian models do exist (jerteh/gpt2-orao, YugoGPT) but the claim here is the fully open stack.

I built this over one summer. The final training run cost ~$230 of Spot H100 time in cloud credits. Out of pocket the whole project was about ~$110.

The model is here to demonstrate the released data pipeline and to be a native-BCS base for finetunes. It writes fluent, native Serbian (Latin/gajica) across registers: news, forums, recipes, casual chat, literary prose. But it's (obviously) not a knowledge base or a competitive assistant.

Specs

Corpus (~23.3B unique tokens)

The corpus is a 96% FineWeb-2 backbone plus deliberate register injection. The pipeline (released alongside) does LID → Cyrillic→Latin transliteration (diacritics preserved; č ć š ž đ are load-bearing) → decluttering → dedup → tokenization.

Use

import torch, json
from safetensors.torch import load_file
from tokenizers import Tokenizer
from model import GPT, GPTConfig   # model.py ships in this repo

cfg = GPTConfig(**json.load(open("config.json"))["config"])
sd = load_file("model.safetensors")
sd["lm_head.weight"] = sd["tok_emb.weight"]     # embeddings are tied, stored once
model = GPT(cfg); model.load_state_dict(sd); model.eval()
tok = Tokenizer.from_file("tokenizer.json")
# sample with temperature 0.7, top-k 200, repetition penalty 1.3 (see sample.py)

This is a base model: it continues text. For chat, use Serblo-125M-Instruct or fine-tune with the released SFT stack.

Limitations

• It makes up specifics. Names, numbers, institutions etc. are fluent inventions ("Beograd, grad sa preko 400 miliona stanovnika").
• It can't faithfully rewrite long passages. Giving it source text to transform; it riffs on the theme instead. Found this by accident while building the eval set and documented it there.
• One rare prompt context (em-dash dialogue openers) makes ~1 in 5 generations start with a stray wrong-script byte before recovering. A repetition penalty of 1.3 and the instruct-tune both do suppress it, but no promises.
• English does suprisingly show up, but often times as authentic BCS code-switching ("Ok so anyway, šta ti se dešava..."), which is corpus-faithful. Standalone English prompts produce word salad.
• Trained on web text with the usual biases, no alignment stage beyond data curation.

Training story

Pretraining ran on Spot H100, checkpointing to GCS every 1,000 steps with an auto-resubmitting supervisor. The run survived three incidents: a plateau false-alarm (actually LR starvation; cutting the cosine schedule 200k→135k gave an annealed finish and val dropped 2.85→2.75), a multi-hour Cloud Logging blackout (checkpoint timestamps were used to verify it was alive) and the cloud project getting billing-suspended hours after the final checkpoint saved. The model was rescued at step 135,000. The details are in the pipeline repo.

Lineage & credits

The synthetic conversational pillar was written by:

Shares are by document count (505,210 total). Gemini carried the volume, the rest is deliberate diversity seasoning: pooling lineages cut phrase repetition 41% below single-model output in balanced tests (self-BLEU 0.331→0.195; the measurement tools ship in the pipeline).

Built by Stefan Selakov (@sterlixlol)