kabyle-stanza-tokenizer

Kabyle Stanza Tokenizer

Sentence tokenizer for Kabyle (Taqbaylit) language, trained on Tatoeba corpus designed to be used on MiniSBD.

Model Details

Files

• kab.onnx — ONNX runtime model
• kab.pt — PyTorch checkpoint
• vocab.json — Character vocabulary (223 entries)
• config.json — Model hyperparameters
• tokenizer_config.json — HF tokenizer config

Usage

With Stanza (PyTorch)

import stanza
nlp = stanza.Pipeline(
    lang="kab", 
    processors="tokenize", 
    tokenize_model_path="kab.pt"
)
doc = nlp("Amcic ha-t-an deg uxxam-nneɣ. Teciḍ fell-as?")
for sent in doc.sentences:
    print(sent.text)
# Amcic ha-t-an deg uxxam-nneɣ.
# Teciḍ fell-as?

With ONNX Runtime

import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession("kab_tokenizer.onnx")
# units: (batch, seq_len) int64 — char IDs from vocab.json
# features: (batch, seq_len, 5) float32 — Stanza features
units = np.zeros((1, 100), dtype=np.int64)
features = np.zeros((1, 100, 5), dtype=np.float32)
outputs = sess.run(None, {"units": units, "features": features})
# outputs[0] shape: (batch, seq_len, 3) — logits for B/I/O

Tokenization Examples

Character Set

The tokenizer handles standard Kabyle Latin characters including:

• ɛ / Ɛ (open e)
• ɣ / Ɣ (voiced velar fricative)
• ṭ, ḍ, č, ǧ (emphatic and palatal consonants)
• Standard ASCII + punctuation

Citation

If you use this model, please cite:

• Tatoeba project: https://tatoeba.org
• Stanza: Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. ACL.

License

Apache-2.0