shahnameh-smollm2-360m

Shahnameh SmolLM2-360M

A full fine-tune of SmolLM2-360M on Shahnameh-style verse. Small enough to run on CPU, no GPU required.

English is the usable language here. Persian and German are poor — SmolLM2-360M is English-centric, which is what motivated the Qwen3 models in this project.

The bug worth knowing about

This model shipped looking broken. Its published samples repeated a single line until the token cap, in all three languages, and the project's notes concluded the checkpoint was worthless.

That was wrong, and the diagnosis is instructive. Teacher-forcing each gold answer and reading the next-token distribution where EOS belongs showed P(EOS) = 0.55, with <|endoftext|> the top-1 prediction in 17 of 30 cases. The model knew exactly where to stop. Greedy decoding simply locked into a self-reinforcing loop and never reached a state where stopping was likely.

No retraining. The weights in this repo carry corrected generation defaults, so a bare generate() call terminates properly.

Usage (CPU)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

mid = "SaeedAngiz1/shahnameh-smollm2-360m"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16,
                                             low_cpu_mem_usage=True)
messages = [
    {"role": "system", "content": "You are a poet who answers in the voice of Ferdowsi's Shahnameh, in English verse."},
    {"role": "user", "content": "Who was Rostam?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, pad_token_id=tok.eos_token_id)   # defaults are already correct
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

Training

SmolLM2-360M, full fine-tune · lr 2e-5 · 3 epochs · 51 steps · 270 examples · response-only loss with EOS supervised · 2× Tesla T4 with DDP.

The distributed job crashed on an NCCL ALLGATHER mismatch after training completed, during export. A separate single-process notebook recovered and exported the preserved checkpoint.

Honest limitations

• Facts are frequently wrong. The dataset teaches voice, not Shahnameh genealogy. Expect invented names and inverted relationships.
• Persian has never been reviewed by a native speaker. Rhyme and script purity are measured mechanically; meter (عروض), grammar and classical register are unverified.
• Output varies run to run at temperature=0.7. Judge on averages, not favourites.
• Trained on only 270 (or 90) examples. That is enough for style, not for prosody.

How it was built

Trained on Kaggle's free tier: 2× NVIDIA Tesla T4. Total compute for the whole project is under three hours. Full method, measurements, charts, and a documented record of four separate failures: https://github.com/SaeedAngiz1/shahnameh-slm

Credits

Ferdowsi's Shahnameh (completed 1010 CE). The training verse is newly written in his style — it is not his text.