LFM2.5-2.6B-Pashto-Zi

LFM2.5-2.6B-Pashto-Zi 🇦🇫

Pashto + Urdu Token Surgery for LiquidAI/LFM2.5-2.6B

This repository contains a tokenizer and embedding-extension version of:

LiquidAI/LFM2.5-2.6B

with additional single-character tokens for Pashto and related Arabic-script characters that were previously missing from the tokenizer.

Model

• Base model: LiquidAI/LFM2.5-2.6B
• Model repository: nassimjp/LFM2.5-2.6B-Pashto-Zi
• Original tokenizer vocabulary: 125,017
• Extended vocabulary: 125,039
• New tokens: 22
• Embedding dimension: 2048
• Embedding dtype during surgery: torch.bfloat16

The original embedding matrix had the shape:

(128000, 2048)

After tokenizer extension and resizing, the model embedding matrices have the shape:

(125039, 2048)

The input and output embeddings share the same storage.

---

🇦🇫 Why Pashto Token Surgery?

Several important Pashto characters were not represented as independent single tokens by the original tokenizer.

For example:

ښ
څ
ځ
ڼ
ږ
ډ
ټ
ړ
ۍ
ې
ګ
ۀ

These characters were previously split into multiple tokenizer IDs.

The purpose of this project is to give these characters their own independent vocabulary entries.

This makes the tokenizer explicitly aware of these characters as individual atoms.

---

🔬 Tokenizer Audit

A total of 81 atoms were audited.

Existing single-token atoms : 59
Atoms requiring new token   : 22

The audit found 22 missing atoms requiring new vocabulary entries.

Examples of previously missing Pashto characters:

ښ
څ
ځ
ڼ
ږ
ډ
ټ
ړ
ۍ
ې
ګ
ۀ

Additional Arabic-script characters and Eastern Arabic numerals were also added.

---

➕ New Vocabulary

The original vocabulary contained:

125017

After adding 22 new tokens:

125039

Exact New Token ID Map

---

🧬 Embedding Initialization

Each of the 22 new vocabulary entries receives its own independent embedding row.

The new embeddings were initialized independently with:

Initializer range: 0.02
Embedding dimension: 2048

The forensic audit confirmed:

Expected shape: (22, 2048)
Actual shape:   (22, 2048)

Each new token therefore owns an independent 2048-dimensional embedding vector.

---

🔎 Embedding Forensics

An exact duplicate search was performed across the new embedding rows.

Result:

NO EXACT DUPLICATE EMBEDDING ROWS

A pairwise cosine similarity analysis was also performed.

The maximum cosine similarity between distinct new-token embeddings was:

0.066502973437

The corresponding pair was:

ٱ  <->  ۴

This confirms that the newly initialized embedding rows are distinct rather than duplicated copies.

---

📊 Embedding Distribution

Original vocabulary

mean      = 0.0003681231
std       = 0.0189890862
norm mean = 0.84232301

New tokens

mean      = -0.0000368604
std       = 0.0198665690
norm mean = 0.89889246

The new-token embeddings were independently initialized and have their own embedding distribution.

---

🔐 Forensic Gate

The final forensic checks confirmed:

✓ Input/output embeddings share storage
✓ Every new atom owns an independent embedding row
✓ No exact duplicate embedding rows
✓ No tokenizer-ID/indexing corruption detected
✓ Embedding matrix successfully resized
✓ Model successfully saved
✓ Tokenizer successfully saved
✓ Model successfully loaded again from Hugging Face

---

🧪 Verification

After uploading the model, it was loaded again from:

nassimjp/LFM2.5-2.6B-Pashto-Zi

The loaded model reported:

Vocab size:
125039

Embedding shape:
torch.Size([125039, 2048])

The newly added tokens were also verified after reloading.

Examples:

ښ -> 125017
څ -> 125018
۹ -> 125038

Existing tokens that were already present in the original tokenizer retain their original IDs.

For example:

ژ -> 84748
پ -> 82992
ک -> 82906
ے -> 82934
۰ -> 85498
۱ -> 85305

These are existing tokenizer entries, not newly added vocabulary entries.

---

🧠 Important Note

This repository represents a tokenizer and embedding vocabulary extension.

The 22 newly added token embeddings were freshly initialized during the vocabulary surgery.

This does not by itself mean that the model has already learned full Pashto language knowledge.

For the newly introduced tokens to acquire meaningful linguistic representations, further training such as continued pretraining / causal language modeling on Pashto data is required.

In other words:

Tokenizer surgery
        ↓
New Pashto atoms
        ↓
Independent embedding rows
        ↓
Pashto continued pretraining
        ↓
Learned Pashto representations

---

🛠️ Intended Use

This model is intended as an experimental foundation for further Pashto language-model development.

Potential next steps include:

• Pashto continued pretraining
• Pashto corpus training
• Pashto causal language modeling
• Pashto instruction tuning
• Pashto SFT
• Pashto tokenizer evaluation
• Pashto token efficiency evaluation
• Pashto language generation experiments

---

📦 Repository Contents

The repository contains the model, tokenizer, configuration, generation configuration, chat template, and forensic report.

Important files include:

config.json
model.safetensors
generation_config.json
tokenizer_config.json
tokenizer.json
chat_template.jinja

pashto_urdu_fresh_embedding_forensics_LiquidAI_LFM2.5-2.6B.json

The forensic report records the tokenizer audit, vocabulary additions, embedding checks, and verification results.

---

📋 Base Model

This project is based on:

LiquidAI/LFM2.5-2.6B

Base model:

Liquid AI — LFM2.5-2.6B

This repository does not claim to reproduce or replace the original base model. It provides an extended tokenizer vocabulary and corresponding resized embedding matrix.

---

🚀 Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "nassimjp/LFM2.5-2.6B-Pashto-Zi"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto"
)

print("Vocabulary:", len(tokenizer))
print("Embedding:", model.get_input_embeddings().weight.shape)

Expected output:

Vocabulary: 125039
Embedding: torch.Size([125039, 2048])

---

🔤 Testing Pashto Characters

You can inspect the newly added atoms with:

pashto_atoms = [
    "ښ", "څ", "ځ", "ڼ", "ږ", "ډ",
    "ټ", "ړ", "ۍ", "ې", "ګ", "ۀ"
]

for char in pashto_atoms:
    token_id = tokenizer.convert_tokens_to_ids(char)
    print(repr(char), "->", token_id)

Expected IDs:

ښ -> 125017
څ -> 125018
ځ -> 125019
ڼ -> 125020
ږ -> 125021
ډ -> 125022
ټ -> 125023
ړ -> 125024
ۍ -> 125025
ې -> 125026
ګ -> 125027
ۀ -> 125028

---

⚠️ Status

Tokenizer surgery: ✅ Complete

Vocabulary resize: ✅ Complete

Embedding forensic audit: ✅ Passed

Model save: ✅ Complete

Tokenizer save: ✅ Complete

Hugging Face upload: ✅ Complete

Reload verification: ✅ Complete

Pashto continued pretraining: ⏳ Not performed as part of this surgery

Pashto SFT: ⏳ Not performed as part of this surgery

---

📜 Forensic Report

A complete forensic report was generated during the surgery:

pashto_urdu_fresh_embedding_forensics_LiquidAI_LFM2.5-2.6B.json

The report contains:

• tokenizer audit
• missing-token inventory
• new token IDs
• embedding shapes
• embedding fingerprints
• duplicate detection
• pairwise cosine analysis
• old/new embedding distribution
• final forensic gate
• model verification

---

👤 Author

nassimjp

Pashto AI / Pashto NLP experiments

---

📄 License

Please refer to the license of the original base model and repository configuration before redistributing modified model weights.

---

⭐ Project

LFM2.5-2.6B-Pashto-Zi

A small but important tokenizer surgery:

LiquidAI/LFM2.5-2.6B
          +
22 new Pashto / Arabic-script atoms
          ↓
125,017 → 125,039 vocabulary
          ↓
Pashto-ready tokenizer foundation

The tokenizer now has dedicated vocabulary entries for the missing Pashto characters.