SemiticGPT-3B

SemiticGPT-3B

A 3.14B parameter multilingual language model trained from scratch for Hebrew, Arabic, Persian (Farsi), and English — a script-diverse, low-resource language cluster centered on Semitic languages.

Model Details

Key Results

Sentiment Classification (v4, clean balanced eval)

Cross-lingual Transfer (Experiment B)

English-only SFT barely transfers to non-English languages, proving multilingual SFT is necessary:

Tokenizer Efficiency (Experiment C)

Our tokenizer uses 49-69% fewer tokens than Llama-2 for Hebrew/Arabic/Farsi:

Files

• base_model.pt — Pretrained base model (no SFT)
• sft_model_v4.pt — Fine-tuned model (v4, sentiment + translation)
• multilingual_32k.model — SentencePiece tokenizer
• config.json — Model configuration
• exp_ab_results.json — Experiment A+B results
• exp_c_tokenizer_ablation.json — Experiment C results

Usage

import torch
import sentencepiece as spm

# Load tokenizer
sp = spm.SentencePieceProcessor('multilingual_32k.model')

# Load model (see model_arch.py for architecture)
from model_arch import GPT
model = GPT()
state = torch.load('sft_model_v4.pt', map_location='cpu', weights_only=True)
model.load_state_dict(state['model_state_dict'])
model.eval()

# Generate
prompt = "<|user|> סווג את הרגש של הטקסט הבא (חיובי/שלילי):\nאני אוהב את הספר הזה!\n<|assistant|> "
ids = sp.encode(prompt)
x = torch.tensor([ids])
with torch.no_grad():
    for _ in range(20):
        logits = model(x)
        next_id = logits[0, -1].argmax().item()
        if next_id == 2: break  # EOS
        x = torch.cat([x, torch.tensor([[next_id]])], dim=1)
print(sp.decode(x[0, len(ids):].tolist()))
# → חיובי

Citation

Paper forthcoming.

License

Apache 2.0