kurdish-roberta

Kurdish Sorani RoBERTa

A RoBERTa language model for Central Kurdish (Sorani), trained from scratch on a 1-million-sentence Kurdish corpus with a custom 32k Kurdish BPE tokenizer. It's a masked-language model meant as a base for fine-tuning on Kurdish tasks (classification, NER, embeddings, and so on).

It's the base model behind haryads/kurdish-sentence-embeddings.

Usage

from transformers import pipeline

fill = pipeline("fill-mask", model="haryads/kurdish-roberta")
fill("زمانی کوردی زۆر <mask> ە")

Or load it directly to fine-tune:

from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained("haryads/kurdish-roberta")
model = AutoModel.from_pretrained("haryads/kurdish-roberta")

Architecture

Training

• Corpus: 1M Sorani sentences
• Tokenizer: custom 32k byte-level BPE for Kurdish
• Objective: masked language modeling (15% masking)

Intended use and limits

A base model, not an instruct or chat model. Best used fine-tuned for a specific Kurdish task. Sorani only, and the corpus is news-heavy, so very colloquial or domain-specific text may need in-domain fine-tuning.

License

Apache-2.0.