gollem-v4-250m-pl

GoLLeM v4 — 250M (Polish, research base model — final checkpoint + training trajectory)

⚠️ STATUS: TRAINING COMPLETE — RESEARCH BASE MODEL. This repository contains the final checkpoint (ckpt_48828, 100% of the planned budget) together with the full series of training snapshots (roughly 4–100% of training), released for research purposes only. This is a base (pretrained) model, not instruction-tuned, and not a production system. It is a 250M research model trained on a single GPU — do not treat it as a general measure of "how good a Polish LM can be".

GoLLeM v4 is a 250M-parameter decoder-only (GPT-style) Polish language model, trained from scratch on a Polish corpus. It is a Slayer Lab project. We publish the full series of snapshots from successive stages of training (through the final checkpoint) so that others can study how the model learns over time.

Author: Arkadiusz Słota (Slayer Lab).

What it is for (and what it is not)

Intended use: research on scaling and training dynamics (comparing successive snapshots), analysis of a Polish language model, and tests of fluency and text continuation in Polish.

Not an instruction-following model. This is a base (pretrained) model, not instruction-tuned: it continues the text you give it, it does not follow commands. It understands Polish only — by design, it performs poorly in English and when asked questions without context. It does not reliably end its output (it does not consistently emit the <|endoftext|> token; at the final checkpoint the measured end-of-sequence rate is ~0, so it generates up to the length limit). This is expected for a base model of this size, not a bug.

Training snapshots (trajectory)

Note on metric-only rows (ckpt_4000, ckpt_44000): their held-out BPB was measured and is shown for trajectory continuity, but their weights are not published — ckpt_4000 is an early snapshot that was never saved to the release series, and ckpt_44000 was rotated out during training. Every other listed snapshot is downloadable.

Training is complete: the final checkpoint is ckpt_48828 (48,828 steps; the final uninterrupted segment took ~71 h on a single RTX 5090, see Training recipe). BPB falls steadily across the run (0.978 → 0.870), flattening to a plateau in the final steps; validation loss tracks training loss closely, so there is no sign of overfitting. The small bumps in validation loss are single-batch noise, not a change in trend. Over the final steps (46000–48828) the held-out BPB has converged to a plateau of ~0.8696–0.8700 — the sub-0.001 variation between these points is evaluation noise (16-window eval), not continued improvement; the model has effectively converged at this token budget. (The final checkpoint fell between evaluation points, so it has no separate validation-loss reading; its held-out BPB was measured directly.)

How we validated the forecast (out of sample). Throughout training we tested a power-law forecast on steps that were not part of the fit:

• The prediction for step 18000 was on target. Between steps 20000 and 26000 the model came in below the extrapolation — it learned faster than the power law predicted.
• From step 26000 onward, results matched the refreshed fit within Δ ≈ −0.002 to −0.004 BPB. The forward check held twelve times in a row (steps 26000–48000), through the second half of the cosine learning-rate decay (there was no separate annealing phase; see Training recipe).
• An anti-collapse diversity monitor (distinct-n on generated text) flagged a transient dip at step 38000 (distinct-3 fell to 0.930). This was a benign annealing transient, not memorization: the direct held-out test excluded memorization — held-out BPB kept falling through the flagged window (0.8807 → 0.8772 → 0.8746) — and distinct-n rebounded. Across the whole run the model is over-determined benign on both axes — the direct held-out discriminator (primary) and the distinct-n proxy (secondary, 0.9564 at the endpoint) — with memorization directly disconfirmed throughout.

End of training (measured, not forecast). The endpoint is no longer an estimate. At the final checkpoint the held-out BPB = 0.8700 (bits-per-byte), measured at step 48828 and recompute-verified independently (two separate evaluation runs — the trainer's and an independent re-computation — produced the same number; the evaluation path was validated by reproducing the step-48000 value exactly). Across the deep tail (steps 46000–48828) the held-out BPB plateaus at ~0.8696–0.8700, essentially flat within evaluation noise. For the record, the earlier power-law forecast (~0.869) was mildly optimistic: the real measurement came in at 0.8700, consistent with the observed flattening. The theoretical floor for the language is around 0.83 BPB.

Architecture

The architecture is custom and does not load via transformers/AutoModel. Use the included loader modeling_gollem.py (standalone; requires only torch, tokenizers and safetensors).

Training data

Corpus: SlayerLab/gollem-corpus-16b-pl — 16.58B tokens, 100% Polish, cleaned (deduplication, PII filtering, decontamination):

Training target: 24B tokens (~1.45 epochs). This is deliberate overtraining — roughly 5× the Chinchilla-optimal token count — which is optimal for the cost of later inference rather than for training itself. Hardware: a single RTX 5090.

Training recipe

Everything below is taken from the training script and the run log of this model; values the logs do not record are marked as not recorded.

Script: train_250m.py in this repository (a copy of the file with sha256 70d53f682b75752f…; local example paths and internal ticket ids in comments removed, and checkpoints loaded with torch.load(..., weights_only=True) when resuming; no other change). One process on one GPU.

Command (paths replaced by placeholders):

torchrun --standalone --nproc_per_node=1 train_250m.py \
  --data <train register files> --val-data <held-out file> \
  --total-tokens 24e9 --batch 480 --accum-steps 20 \
  --ckpt-dir <ckpt dir> --log-dir <log dir> \
  --ckpt-every 2000 --log-every 50 --val-every 500 --run-id v4-fire

Defaults used by this command: --lr 3e-4, --wd 0.1, --warmup-frac 0.01, --val-batches 16, --seed 1337, --keep-ckpt 3.

What happened during the run (for reproducibility):

• The first attempt crashed twice (SIGBUS) around steps 2,000–3,650. The released run resumed from the step-2,000 checkpoint on the original data and ran without interruption to the end. A short restart on a different data build (steps 2,000–2,300) was discarded and is not part of this model.
• Random-number state is not saved in checkpoints; on resume the seed is set again, so the data-window order after step 2,000 partly repeats the order from the start of training.
• The training split size logged by the trainer (16,514,371,180 tokens) is 19.2M tokens larger than the train count on the corpus card; the reason is not recorded.

Usage

pip install torch tokenizers safetensors
# final checkpoint (default):
python generate.py "Polska to kraj" --n 100 --temp 0.8 --topk 50
# a specific snapshot from the trajectory:
python generate.py "Polska to kraj" --ckpt ckpt_6000.safetensors --n 100

from modeling_gollem import GollemGPT
m = GollemGPT.from_pretrained("./")                          # final checkpoint
# m = GollemGPT.from_pretrained("./", ckpt="ckpt_6000.safetensors")  # a specific stage
print(m.generate("Stolica Polski to", max_new_tokens=100, temperature=0.8, top_k=50))

Files in the repository: ckpt_*.safetensors (weights of each snapshot, pickle-free format), config.json (architecture and list of snapshots), tokenizer.json (V32k), modeling_gollem.py and generate.py (standalone loader and generation script), train_250m.py (training script, see Training recipe).

Note on format: weights are stored as safetensors (pure tensors, no executable code), which is safer for public release than raw .pt (pickle) files. Full training checkpoints (with optimizer state, for resuming training) remain internal.

Limitations and ethical notes

• This is a 250M research base model; although training is complete, it is small and is not a production or instruction-following system.
• It was trained on web data; despite filtering, it may reproduce biases, errors and content present in the corpus.
• A PII filter was applied (residual rate ≈ 0.0018%), but this does not guarantee the complete absence of personal data.
• Research use only. Slayer Lab accepts no responsibility for production use.

License and data provenance

Weights: Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0). You may use, adapt and publish derivatives (including fine-tunes) under the terms of this licence, provided you give attribution (GoLLeM v4, Arkadiusz Słota / Fabryka AI, link to this repository) and release derivative weights under the same licence.

Training data: SlayerLab/gollem-corpus-16b-pl. Partitions default and cc-by-sa contain only apache-2.0, CC-BY, CC-BY-SA and official documents, with no non-commercial records. Partition web (HPLT v3 PL, ~88 % of tokens) is used on the text-and-data-mining basis described on the corpus card; its license column shows CC0, which is the licence of the HPLT compilation, not of the texts. News items carry the dataset licence; the articles remain under their authors' copyright. This is our description of the data, not legal advice.

---

GoLLeM v4 — Slayer Lab. Author: Arkadiusz Słota. Research base model (final checkpoint + full training trajectory); card maintained by the Slayer team.