nanogpt-srpske-epske-pesme

nanoGPT Srpske Epske Pesme

This is a small character-level nanoGPT checkpoint trained on Serbian epic folk poetry (tiny_srpske_epske_pesme).

It is based on karpathy/nanoGPT and was trained on alkibijad/tiny_srpske_epske_pesme.

Опис на српском 🇷🇸

Мали nanoGPT модел трениран на корпусу српских епских народних песама. Ово је ”играчка” модел а не прави велики језички модел.

Example Outputs

Example output data is included in this folder. Sample:

И он шета од 'ванаест година,
Док ево ти свога сокола,
И он сједе потеже сивијети,
А на њиме три товара блага,
Која сједи сокола добра коња,
Сваки соколе чадор одјавио,
Те разљуби прихвати љубу:
Што ј' у њега три товара блага,
Држи синџир-товара блага,

Usage

Install dependencies:

pip install -r requirements.txt

Minimal Python usage:

import json
import torch
from model import GPTConfig, GPT

# Load config
with open("config.json") as f:
    cfg = json.load(f)

# Build model
conf = GPTConfig(
    vocab_size=cfg["vocab_size"],
    block_size=cfg["block_size"],
    n_layer=cfg["n_layer"],
    n_head=cfg["n_head"],
    n_embd=cfg["n_embd"],
    dropout=cfg["dropout"],
    bias=cfg["bias"],
)
model = GPT(conf)
model.load_state_dict(torch.load("pytorch_model.bin", map_location="cpu"))
model.eval()

# Tokenize and generate
from char_tokenizer import encode, decode

prompt = "Марко"
ids = torch.tensor([encode(prompt)], dtype=torch.long)
out = model.generate(ids, max_new_tokens=200, temperature=0.8, top_k=40)
print(decode(out[0].tolist()))

Generate ten sample files on CPU:

python scripts/generate_srpske_epske_pesme_outputs.py --device cpu

The default prompt is a single newline character ("\n"). The script uses it as a short conditioning prefix, then removes it before writing the generated text files.

You can provide a custom prompt:

python scripts/generate_srpske_epske_pesme_outputs.py --start "Марко" --device cpu

Model Details

• Architecture: nanoGPT character-level GPT
• Layers: 6
• Heads: 6
• Embedding size: 384
• Context length: 256 characters
• Vocabulary size: 157 characters
• Dropout: 0.2
• Checkpoint iteration: 1450
• Best validation loss: 1.4569684267044067
• Checkpoint size: about 123 MB

Training Data

The local training dataset contains 737,054 Unicode characters of Serbian Cyrillic epic poetry and is published as alkibijad/tiny_srpske_epske_pesme.

The raw training text is not included in this model repository. Only the character vocabulary metadata needed for decoding and sampling is included.

Files

• out-tiny-srpske-epske-pesme/ckpt.pt: trained PyTorch checkpoint
• pytorch_model.bin: model state_dict for the simpler loading example
• config.json: model architecture config for the simpler loading example
• char_tokenizer.py: small character tokenizer helper
• data/tiny_srpske_epske_pesme/meta.pkl: character vocabulary metadata
• config/train_srpske_epske_pesme.py: training configuration
• scripts/generate_srpske_epske_pesme_outputs.py: generation script
• model.py: nanoGPT model definition needed by the generation script
• generated/srpske_epske_pesme/: example output samples generated from this checkpoint
• requirements.txt: minimal runtime dependencies
• LICENSE: original nanoGPT MIT license for the included code

The Hub also includes .gitattributes, which is generated by Hugging Face for Git/LFS file handling.

Limitations

This is a tiny character-level model trained on a narrow poetry corpus. It can produce text that resembles the form of the training data, but it does not have general language understanding, factual reliability, or chat behavior.