Downloads · 30 days
0
zustmartin/slovenian-kenlm-5gram-mixed
slovenian-kenlm-5gram-mixed is a machine learning model from zustmartin. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for kenlm. The card lists the license as cc-by-sa-4.0.
A versatile 5-gram language model for Slovenian trained on Wikipedia, forum discussions, and social media data using KenLM.
Downloads · 30 days
0
Access
Public
Updated Mar 11, 2026
Repo size
26.2 MB
Likes
0
Public
Click a slice to open those files.
.binary26.2 MB · 100%
From the Hugging Face model README
A versatile 5-gram language model for Slovenian trained on Wikipedia, forum discussions, and social media data using KenLM.
This is a statistical n-gram language model designed for perplexity-based filtering and quality assessment of Slovenian text. The model was trained using the KenLM toolkit on a diverse mix of formal and informal Slovenian text sources.
This model is primarily intended for:
Unlike models trained only on Wikipedia, this mixed model handles:
The model can be used to filter large text corpora by computing perplexity scores and keeping only texts within a reasonable range (25-5000), effectively removing:
Examples of perplexity scores on different text types:
| Text Type | Example | Perplexity |
|---|---|---|
| Formal/encyclopedic | "V Sloveniji živi približno dva milijona ljudi." | 44 |
| Forum question | "Ima kdo izkušnje s tem programom." | 74 |
| Everyday conversation | "Danes je lep sončen dan." | 318 |
| Common slang | "Ma dej no to ni resno." | 416 |
| Informal praise | "Haha to je bilo smešno res." | 1,523 |
| English text | "the quick brown fox jumps" | 37,960 |
| Gibberish | "asdf qwer zxcv tyui hjkl" | 408,383 |
import kenlm
from huggingface_hub import hf_hub_download
# Download model from HuggingFace
model_path = hf_hub_download(
repo_id="zustmartin/slovenian-kenlm-5gram-mixed",
filename="slovenian_5gram_mixed.binary"
)
# Load the model
model = kenlm.Model(model_path)
# Compute perplexity for a text
text = "to je primer slovenskega besedila"
perplexity = model.perplexity(text.lower())
print(f"Perplexity: {perplexity}")
# Score individual sentences
score = model.score(text.lower())
print(f"Log10 probability: {score}")
# Filter a corpus
def is_good_slovenian(text, lower=25, upper=5000):
"""Check if text is good quality Slovenian."""
ppl = model.perplexity(text.lower().strip())
return lower <= ppl <= upper
texts = [
"Danes je lep sončen dan.", # Good
"asdf qwer zxcv", # Bad (gibberish)
"the quick brown fox", # Bad (English)
]
for text in texts:
ppl = model.perplexity(text.lower())
status = "✅ KEEP" if is_good_slovenian(text) else "❌ FILTER"
print(f"{status} - {text} (perplexity: {ppl:.0f})")
Three diverse sources:
Wikipedia (formal, encyclopedic)
wikimedia/wikipedia dataset, version 20231101.sl)Janes-Forum (informal healthcare forums)
Janes-Tag (social media, tweets, blogs)
Processing Pipeline:
a-z, čšžćđ, digits, and basic punctuationThe model was trained using KenLM with pruning for smaller model size:
--prune 0 1 1 2 2 (removes rare n-grams)lmplz -o 5 --prune 0 1 1 2 2 < slovenian_mixed_for_kenlm.txt > slovenian_5gram_mixed.arpa
build_binary slovenian_5gram_mixed.arpa slovenian_5gram_mixed.binary
slovenian_5gram_mixed.binary - Binary KenLM model (recommended for production use)To use this model, install the KenLM Python bindings:
pip install https://github.com/kpu/kenlm/archive/master.zip
Please cite the training data sources:
@misc{janesTag,
title={Training corpus Janes-Tag 3.0},
author={Erjavec, Toma{\v z} and Fišer, Darja and Krek, Simon and Ledinek, Nina and Arhar Holdt, {\v S}pela},
url={http://hdl.handle.net/11356/1732},
note={Slovenian language resource repository {CLARIN}.{SI}},
year={2022}
}
@misc{janesForum,
title={Janes-Forum 1.0: Corpus of Slovene forum posts},
author={Fišer, Darja and Erjavec, Toma{\v z} and Ljubešić, Nikola},
url={http://hdl.handle.net/11356/1139},
note={Slovenian language resource repository {CLARIN}.{SI}},
year={2017}
}
The model weights are released under CC-BY-SA 4.0, compatible with the training data licenses:
Tested on diverse Slovenian text types with perplexity scores ranging from:
Recommended filtering threshold for web-scraped corpora: 25-5,000