lv_spacy_cnn_5pct

Latvian SpaCy Model: lv_spaCy_CNN

GitHub Repo

https://github.com/JesseLau24/Fengdi_Huang_Master_Thesis_Repo

---

Overview

This is a spaCy pipeline for Latvian, built with the spaCy Tok2Vec CNN backbone.

It includes the following components:

• Tok2Vec (spaCy Tok2Vec CNN)
• Tagger
• Morphologizer
• Parser
• Sentence Segmenter (senter)
• Lemmatizer

Model type: spaCy CNN Pipeline Language: Latvian (lv)
Recommended hardware: CPU for small-scale use, GPU recommended for faster inference.

---

Training Data

The model was trained on the Latvian UD Treebank v2.16, which is derived from the Latvian Treebank (LVTB) created at the University of Latvia, Institute of Mathematics and Computer Science, Artificial Intelligence Laboratory (AI Lab).

• Dataset source: UD Latvian LVTB
• License: CC BY-SA 4.0
• Data splits:
  • Train: 15,055 sentences
  • Dev: 2,080 sentences
  • Test: 2,396 sentences

---

Acknowledgements

• Thanks to the University of Latvia, AI Lab, and all contributors of the Latvian UD Treebank.
• Inspired by the spaCy ecosystem and training framework.
• The Latvian UD Treebank was developed with support from multiple grants, including:
  • European Regional Development Fund (Grant No. 1.1.1.1/16/A/219, 1.1.1.2/VIAA/1/16/188)
  • State Research Programme "National Identity"
  • State Research Programme "Digital Resources for the Humanities" (Grant No. VPP-IZM-DH-2020/1-0001)
  • State Research Programme "Research on Modern Latvian Language and Development of Language Technology" (Grant No. VPP-LETONIKA-2021/1-0006)

---

Special Thanks

Special Thanks to all contributors who participated in the Beta test and espically those who provided valuable feedback

License

This model is released under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0).

You are free to:

• Share — copy and redistribute the material in any medium or format, for any purpose, even commercially.
• Adapt — remix, transform, and build upon the material for any purpose, even commercially.

Under the following terms:

• Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made.
• ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.

---

References

• Pretkalniņa, L., Rituma, L., Saulīte, B., et al. (2016–2025). Universal Dependencies Latvian Treebank (LVTB).
• Grūzītis, N., Znotiņš, A., Nešpore-Bērzkalne, G., Paikens, P., et al. (2018). Creation of a Balanced State-of-the-Art Multilayer Corpus for NLU. LREC 2018.
• Pretkalniņa, L., Rituma, L., Saulīte, B. (2016). Universal Dependency Treebank for Latvian: A Pilot. Baltic Perspective Workshop.

---

---

Usage

You can:

Download the model directly from the Hugging Face Hub
Using huggingface_hub.snapshot_download, the model files will be automatically fetched and cached locally.

  ```python
  import spacy
  from huggingface_hub import snapshot_download
  
  # Load the pipeline
  model_dir = snapshot_download(repo_id="JesseHuang922/lv_spacy_cnn_5pct", repo_type="model")
  nlp = spacy.load(model_dir)
  ```

   

---

Dependencies

The following Python packages are required to run the Latvian spaCy pipeline:

Optional but recommended

Install all dependencies with just one command line:

pip install \
spacy>=3.8.7 \
spacy-transformers>=1.3.9 \
transformers>=4.49.0 \
torch>=2.8.0 \
tokenizers>=0.21.4 \
safetensors>=0.6.2 \
huggingface-hub>=0.34.4 \
hf-xet>=1.1.10

Example Code

import spacy
import numpy as np
from huggingface_hub import snapshot_download

# Load the pipeline
model_dir = snapshot_download(repo_id="JesseHuang922/lv_spacy_cnn_5pct", repo_type="model")
nlp = spacy.load(model_dir)

# Example text
text = """Baltijas jūras nosaukums ir devis nosaukumu baltu valodām un Baltijas valstīm.
Terminu "Baltijas jūra" (Mare Balticum) pirmoreiz lietoja vācu hronists Brēmenes Ādams 11. gadsimtā."""

# Process text
doc = nlp(text)

# ------------------------
# Tokenization 
# ------------------------
print("Tokens:")
print([token.text for token in doc])

# ------------------------
# Lemmatization
# ------------------------
print("Lemmas:")
print([token.lemma_ for token in doc])

# ------------------------
# Part-of-Speech Tagging
# ------------------------
print("POS tags:")
for token in doc:
    print(f"{token.text}: {token.pos_} ({token.tag_})")

# ------------------------
# Morphological Features
# ------------------------
print("Morphological features:")
for token in doc:
    print(f"{token.text}: {token.morph}")

# ------------------------
# Dependency Parsing
# ------------------------
print("Dependency parsing:")
for token in doc:
    print(f"{token.text} <--{token.dep_}-- {token.head.text}")

# ------------------------
# Sentence Segmentation
# ------------------------
print("Sentences:")
for sent in doc.sents:
    print(sent.text)

# ------------------------
# Check Pipeline Components
# ------------------------
print("Pipeline components:")
print(nlp.pipe_names)

# Transformer vectors
vectors = np.vstack([token.vector for token in doc])
print("Token vectors shape:", vectors.shape)