LFM2.5-2.6B-Pashto-Zi-b

LFM2.5-2.6B-Pashto-Zi 🇦🇫

Pashto + Urdu + Persian + Balochi + Sindhi Token Surgery for LiquidAI/LFM2.5-2.6B

This repository contains a tokenizer and embedding-extension version of:

LiquidAI/LFM2.5-2.6B

with additional single-character tokens for Pashto, Sindhi, Balochi, and related Arabic-script characters that were previously missing from the tokenizer.

Model

• Base model: LiquidAI/LFM2.5-2.6B
• Model repository: nassimjp/LFM2.5-2.6B-Pashto-Zi-b
• Original tokenizer vocabulary: 125,017
• Extended vocabulary: 125,073
• New tokens: 56
• Embedding dimension: 2048
• Embedding dtype during surgery: torch.bfloat16

The original embedding matrix had the shape:

(128000, 2048)

After tokenizer extension and resizing, the model embedding matrices have the shape:

(125073, 2048)

The input and output embeddings share the same storage.

---

🇦🇫 Why Pashto Token Surgery?

Several important Pashto characters were not represented as independent single tokens by the original tokenizer.

For example:

ښ
څ
ځ
ڼ
ږ
ډ
ټ
ړ
ۍ
ې
ګ
ۀ

These characters were previously split into multiple tokenizer IDs.

The purpose of this project is to give these characters their own independent vocabulary entries.

This makes the tokenizer explicitly aware of these characters as individual atoms.

---

🌏 Extended Coverage: Sindhi, Balochi, and More

The same problem exists for other Arabic-script languages. This surgery adds dedicated single-character tokens for:

• Sindhi (52-letter Perso-Arabic alphabet): ٻ ڀ ٿ ٽ ٺ ڄ ڃ ڇ ڌ ڏ ڊ ڍ ڙ ڦ ڻ ڱ ڳ ڪ
• Balochi vowels: ۆ ۇ ۈ ۉ ۊ ۋ ۏ
• Arabic-script hamza / heh variants: ٱ ۀ ە ۃ ۂ
• Extended Arabic-Indic digits: ۴ ۵ ۶ ۷ ۸ ۹
• Bidi controls: U+202A – U+202E (LRE, RLE, PDF, LRO, RLO)
• Additional harakat: ٓ ٔ ٕ
• Arabic symbol: ؉

---

🔬 Tokenizer Audit

A total of 175 unique atoms were audited (181 raw entries, with intentional duplicates kept for a separate "push-all" script).

Existing single-token atoms : 119
Atoms requiring new token   :  56

The audit found 56 missing atoms requiring new vocabulary entries.

Examples of previously missing Pashto characters:

ښ
څ
ځ
ڼ
ږ
ډ
ټ
ړ
ۍ
ې
ګ
ۀ

Additional Sindhi, Balochi, Arabic-script, bidi-control, and harakat characters were also added.

---

➕ New Vocabulary

The original vocabulary contained:

125017

After adding 56 new tokens:

125073

Exact New Token ID Map

---

🧬 Embedding Initialization

Each of the 56 new vocabulary entries receives its own independent embedding row.

The new embeddings were initialized independently with:

Initializer range: 0.02
Embedding dimension: 2048

The forensic audit confirmed:

Expected shape: (56, 2048)
Actual shape:   (56, 2048)

Each new token therefore owns an independent 2048-dimensional embedding vector.

---

🔎 Embedding Forensics

An exact duplicate search was performed across the new embedding rows.

Result:

NO EXACT DUPLICATE EMBEDDING ROWS

A pairwise cosine similarity analysis was also performed.

The maximum cosine similarity between distinct new-token embeddings was:

0.092469498515

The corresponding pair was:

ۈ  <->  U+202C

This confirms that the newly initialized embedding rows are distinct rather than duplicated copies.

---

📊 Embedding Distribution

Original vocabulary

mean      = 0.0003681231
std       = 0.0189890880
norm mean = 0.84232318

New tokens

mean      = -0.0000251698
std       = 0.0200549196
norm mean = 0.90741420

The new-token embeddings were independently initialized and have their own embedding distribution.

---

🔐 Forensic Gate

The final forensic checks confirmed:

✓ Input/output embeddings share storage
✓ Every new atom owns an independent embedding row
✓ No exact duplicate embedding rows
✓ No tokenizer-ID/indexing corruption detected
✓ Embedding matrix successfully resized
✓ Model successfully saved
✓ Tokenizer successfully saved
✓ Model successfully loaded again from Hugging Face

---

🧪 Verification

After uploading the model, it was loaded again from:

nassimjp/LFM2.5-2.6B-Pashto-Zi-b

The loaded model reported:

Vocab size:
125073

Embedding shape:
torch.Size([125073, 2048])

The newly added tokens were also verified after reloading.

Examples:

ښ -> 125019
څ -> 125017
۹ -> 125063
ٻ -> 125028
ۆ -> 125046

Existing tokens that were already present in the original tokenizer retain their original IDs.

For example:

ی -> 40443
ژ -> 84748
پ -> 82992
ک -> 82906
ے -> 82934
۰ -> 85498
۱ -> 85305
۲ -> 86281
۳ -> 87526

These are existing tokenizer entries, not newly added vocabulary entries.

---

🧠 Important Note

This repository represents a tokenizer and embedding vocabulary extension.

The 56 newly added token embeddings were freshly initialized during the vocabulary surgery.

This does not by itself mean that the model has already learned full Pashto (or Sindhi, Balochi, …) language knowledge.

For the newly introduced tokens to acquire meaningful linguistic representations, further training such as continued pretraining / causal language modeling on multilingual Arabic-script data is required.

In other words:

Tokenizer surgery
        ↓
New Pashto + Sindhi + Balochi atoms
        ↓
Independent embedding rows
        ↓
Continued pretraining
        ↓
Learned representations

---

🛠️ Intended Use

This model is intended as an experimental foundation for further Arabic-script language-model development.

Potential next steps include:

• Pashto continued pretraining
• Urdu continued pretraining
• Sindhi continued pretraining
• Balochi continued pretraining
• Multilingual Arabic-script causal language modeling
• Instruction tuning / SFT
• Tokenizer evaluation
• Token efficiency evaluation
• Language generation experiments

---

📦 Repository Contents

The repository contains the model, tokenizer, configuration, generation configuration, chat template, and forensic report.

Important files include:

config.json
model.safetensors
generation_config.json
tokenizer_config.json
tokenizer.json
chat_template.jinja

pashto_urdu_fresh_embedding_forensics_LiquidAI_LFM2.5-2.6B.json

The forensic report records the tokenizer audit, vocabulary additions, embedding checks, and verification results.

---

📋 Base Model

This project is based on:

LiquidAI/LFM2.5-2.6B

Base model:

Liquid AI — LFM2.5-2.6B

This repository does not claim to reproduce or replace the original base model. It provides an extended tokenizer vocabulary and corresponding resized embedding matrix.

---

🚀 Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "nassimjp/LFM2.5-2.6B-Pashto-Zi-b"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto"
)

print("Vocabulary:", len(tokenizer))
print("Embedding:", model.get_input_embeddings().weight.shape)

Expected output:

Vocabulary: 125073
Embedding: torch.Size([125073, 2048])

---

🔤 Testing Newly Added Characters

You can inspect the newly added atoms with:

new_atoms = [
    # Pashto
    "ښ", "څ", "ځ", "ڼ", "ږ", "ډ",
    "ټ", "ړ", "ۍ", "ې", "ګ", "ۀ",
    # Sindhi
    "ٻ", "ڀ", "ٿ", "ٽ", "ٺ", "ڄ", "ڃ", "ڇ",
    "ڌ", "ڏ", "ڊ", "ڍ", "ڙ", "ڦ", "ڻ", "ڱ",
    "ڳ", "ڪ",
    # Balochi
    "ۆ", "ۇ", "ۈ", "ۉ", "ۊ", "ۋ", "ۏ",
    # Hamza / heh variants
    "ٱ", "ۀ", "ە", "ۃ", "ۂ",
    # Extended digits
    "۴", "۵", "۶", "۷", "۸", "۹",
    # Symbols / bidi / harakat
    "؉",
    "\u202a", "\u202b", "\u202c", "\u202d", "\u202e",
    "ٓ", "ٔ", "ٕ",
]

for char in new_atoms:
    token_id = tokenizer.convert_tokens_to_ids(char)
    print(repr(char), "->", token_id)

Expected IDs run from 125017 through 125072 (see the table above).

---

⚠️ Status

Tokenizer surgery: ✅ Complete

Vocabulary resize: ✅ Complete

Embedding forensic audit: ✅ Passed

Model save: ✅ Complete

Tokenizer save: ✅ Complete

Hugging Face upload: ✅ Complete

Reload verification: ✅ Complete

Continued pretraining: ⏳ Not performed as part of this surgery

SFT / instruction tuning: ⏳ Not performed as part of this surgery

---

📜 Forensic Report

A complete forensic report was generated during the surgery:

pashto_urdu_fresh_embedding_forensics_LiquidAI_LFM2.5-2.6B.json

The report contains:

• tokenizer audit
• missing-token inventory
• new token IDs
• embedding shapes
• embedding fingerprints
• duplicate detection
• pairwise cosine analysis
• old/new embedding distribution
• final forensic gate
• model verification

---

👤 Author

nassimjp

Pashto AI / Pashto NLP experiments

---

📄 License

Please refer to the license of the original base model and repository configuration before redistributing modified model weights.

---

⭐ Project

LFM2.5-2.6B-Pashto-Zi-b

A tokenizer surgery extended beyond Pashto:

LiquidAI/LFM2.5-2.6B
          +
56 new Pashto + Sindhi + Balochi + Arabic-script atoms
          ↓
125,017 → 125,073 vocabulary
          ↓
Multilingual Arabic-script tokenizer foundation

The tokenizer now has dedicated vocabulary entries for the missing Pashto, Sindhi, Balochi, and related Arabic-script characters.

---