albert-base-v2-persian-pos-tagger

ALBERT Persian POS Tagger

Part-of-speech tagger for Persian using the 17-tag Universal Dependencies scheme. Fine-tuned from shekar-ai/albert-base-v2-persian-zwnj-naab-mlm on the Persian Universal Dependency Treebank (PerDT).

Tags: ADJ, ADP, ADV, AUX, CCONJ, DET, INTJ, NOUN, NUM, PART, PRON, PROPN, PUNCT, SCONJ, VERB, X, _.

Usage

Shekar

from shekar import POSTagger

tagger = POSTagger()

for word, tag in tagger("کتاب‌ها دریچه‌ای به جهان دانش هستند."):
    print(f"{word}\t{tag}")

کتاب‌ها	NOUN
دریچه‌ای	NOUN
به	ADP
جهان	NOUN
دانش	NOUN
هستند	VERB
.	PUNCT

Transformers

The model was trained on word-level input with labels on the first subtoken of each word and no special tokens. Feeding raw text to the generic token-classification pipeline therefore produces subword-level noise — tokenize word by word and read the prediction of each word's first subtoken:

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

repo = "shekar-ai/albert-base-v2-persian-pos-tagger"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo).eval()

def tag(words):
    input_ids, word_ids = [], []
    for index, word in enumerate(words):
        subtokens = tokenizer.encode(word, add_special_tokens=False) or [tokenizer.unk_token_id]
        input_ids.extend(subtokens)
        word_ids.extend([index] * len(subtokens))

    with torch.no_grad():
        logits = model(input_ids=torch.tensor([input_ids])).logits[0]
    predictions = logits.argmax(-1).tolist()

    tags, previous = [], None
    for word_index, prediction in zip(word_ids, predictions):
        if word_index != previous:
            tags.append((words[word_index], model.config.id2label[prediction]))
            previous = word_index
    return tags

from shekar import WordTokenizer

for word, pos in tag(list(WordTokenizer()("شاهنامه اثر فردوسی در توس سروده شد."))):
    print(f"{word}\t{pos}")

شاهنامه	PROPN
اثر	NOUN
فردوسی	PROPN
در	ADP
توس	PROPN
سروده	VERB
شد	VERB
.	PUNCT

Training

Fine-tuned on the PerDT treebank: 26,196 training, 1,456 development, and 1,455 test sentences. Each word is tokenized independently; the gold tag is assigned to its first subtoken and remaining subtokens are masked out of the loss with -100. The checkpoint with the best macro-F1 was kept.

Evaluation

Measured on the PerDT test split (1,455 sentences, 24,429 tokens):

Per-tag scores:

Macro F1 is held down by the rare tags: INTJ (27 tokens) and X (1 token) contribute as much to the average as NOUN (8,219 tokens). The main substantive weakness is the NOUN/PROPN boundary, where Persian offers no capitalization cue.

<details> <summary>Per-epoch validation during training</summary> </details>

Citation

@article{Amirivojdan2025Shekar,
  author  = {Amirivojdan, Ahmad},
  title   = {{Shekar: A Python Toolkit for Persian Natural Language Processing}},
  journal = {Journal of Open Source Software},
  volume  = {10},
  number  = {114},
  pages   = {9128},
  year    = {2025},
  doi     = {10.21105/joss.09128},
  url     = {https://joss.theoj.org/papers/10.21105/joss.09128}
}