fa_core_news_sm

fa_core_news_sm

Persian pipeline optimized for CPU. Components: tok2vec, tagger, morphologizer, trainable_lemmatizer, parser, ner. Entity labels: PER, LOC, ORG, DAT, MON, TIM, PCT.

Install

pip install https://huggingface.co/Phazel/fa_core_news_sm/resolve/main/fa_core_news_sm-3.8.0-py3-none-any.whl

Use

import spacy

nlp = spacy.load("fa_core_news_sm")
doc = nlp("شرکت ایران خودرو اعلام کرد که تولید خود را افزایش می‌دهد.")

print([(t.text, t.pos_, t.tag_, t.lemma_, t.dep_) for t in doc])
print([(e.text, e.label_) for e in doc.ents])

Accuracy

Scored with spacy benchmark accuracy on the held-out PerDT test split.

Per entity label:

Throughput

Median of repeated nlp.pipe passes over the 146-document PerDT test split (23,825 tokens), timing the pipe only. Warmup pass discarded.

Other packages in this family

Tiers: sm hash embeddings, no vectors, md 50k x 300d floret vectors, lg 200k x 300d floret vectors, trf fine-tuned transformer, GPU recommended.

Standalone vector tables, usable as --paths.vectors for your own training:

Training scripts, configs and evaluation: https://github.com/Fazel94/spacy-persian.

Sources

Notes

Trained on UD_Persian-PerDT, licensed CC BY-SA 4.0; this pipeline is therefore distributed under CC BY-SA 4.0 with attribution to the treebank authors. The ner component is trained on the NER layer shipped in UD_Persian-PerDT's not-to-release/ directory, so it shares the treebank's genre, tokenization and licence. Those labels are SILVER: the treebank README states they were produced by the BERT-based Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. They were transferred onto this pipeline's tokenization by difflib alignment at a 99.86% transfer rate (scripts/transfer_perdt_ner.py); spans that could not be aligned exactly were dropped rather than guessed. Labels PER, LOC, ORG and DAT have 1,300 or more training examples each; MON (205), TIM (135) and PCT (121) are thin and their scores in performance.ents_per_type should be read before relying on them. Multiword tokens (pronominal clitics, enclitic copulas) were merged with spacy convert --merge-subtokens, so a small number of XPOS tags are composite (e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space. doc.noun_chunks under-fires on this pipeline: spacy/lang/fa/syntax_iterators.py matches ClearNLP labels that do not exist in Universal Dependencies, see docs/upstream/fa-noun-chunks.md.