Vesemir-1.5B

Vesemir 1.5B

A model extending the Cirilla family, Vesemir 1.5B is a small language model.

To learn more about Cirilla visit my GitHub Repository.

🤗 compatibility

In contrary to the Cirilla family of models, Vesemir is fully compatible with the huggingface transformers library.

Model overview

Key Features

Vesemir 1.5B is based on Motif and Ministral.

It consists of the following components:

• Differential Attention: each attention layer computes two separate softmax attention maps over split query/key heads and subtracts them (weighted by a learned λ), following Differential Transformer (Ye et al., 2024) — designed to cancel attention noise and produce sparser, more focused attention patterns than standard softmax attention.
• Grouped-Query Attention (GQA): 16 query heads share 8 key/value head groups, cutting KV-cache size and memory bandwidth at inference.
• SwiGLU MLP: feed-forward blocks with RMSNorm pre-normalization.
• (optional) Sliding Window Attention: Allows the model to handle sequences effectively by limiting the attention scope, reducing memory usage.
• License: released under the MIT License.

Training Data

The training data consists of three parts: Pretraining, SFT, DPO

The model was trained on a single A100 40GiB.

General Pretraining

Trained from a random initialization on AnthonyPa57/Plain-text-pretraining — 280k documents (~0.2B tokens) for 1 epoch. The dataset was prepared as a mix of topics in Polish and English.

The dataset consists of 280,107 rows (132,535 EN / 147,572 PL), by source it splits into General 36.8% (TinyStories, PolishStories, Wikipedia), Science 31.4% (fineweb-edu, fineweb-2-pol, rafalposwiata/plsc), Legal 22.9% (Multi Legal Pile), and Witcher 8.9% (witcher fandom + books summary sites) — see the data composition chart above.

Final eval: loss 2.182

Supervised Fine-Tuning (SFT)

The model was instruction-tuned on a mixture of AnthonyPa57/Instruct-training, Instruct-rag-training, Instruct-python-training, and Instruct-translation-training — roughly 1.5M entries (~0.3B tokens total) trained for 1 epoch.

Instruct-training itself blends in five AnthonyPa57/Witcher-* sources, SFT breaks down by data type as Instruct (general) 48.1%, Translation 29.2%, Witcher 9.6%, Python 7.9%, RAG 5.1%.

Final eval: loss 2.41.

Direct Preference Optimization (DPO)

Aligned on AnthonyPa57/DPO-training — around 175k entries trained for 1 epoch.

Reward accuracy rose from ~0.51 (near-random-choice; ln 2 ≈ 0.693) in the first steps to ~0.59 average over the final steps of training.

Final eval: loss 0.6633

Training Overview

The dips in learning rates for pretraining and SFT are a result of manual tweaks, as initially the chosen learning rate values were too high.

Repository Layout

The published Hub repo carries the final (post-DPO) weights at the root, plus each stage's own final checkpoint in its own subfolder:

AnthonyPa57/Vesemir-1.5B
├── model.safetensors, config.json, ...   # final weights (= dpo/)
├── pretraining/                          # end of pretrain
├── sft/                                  # end of SFT
└── dpo/                                  # end of DPO (same as root)

Usage

Vesemir-1.5B uses a custom transformers architecture, so it loads directly through AutoModelForCausalLM / AutoTokenizer — no separate package required, just trust_remote_code=True.

for more information see

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AnthonyPa57/Vesemir-1.5B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Who is Geralt of Rivia, and what is a witcher?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

# Geralt of Rivia is a witcher, a monster hunter who uses magic, alchemy, and enhanced physical abilities gained through mutation.

To load an earlier stage's checkpoint (e.g. the SFT model before DPO alignment) instead of the final one, pass subfolder:

model = AutoModelForCausalLM.from_pretrained(model_id, subfolder="sft", trust_remote_code=True)

Closing Notes

The model was trained on a relatively tight budget of "only" ~200h on A100 40GiB and far from the Chinchilla optimal amount of training data (~20 per model parameter, for 1.5B parameters giving 30B training tokens), hence it should be considered as something of a base model for further development.