amharic-50m-architecture-benchmark

50M-Class Comparative Architecture Benchmark (Amharic)

This repository contains trained checkpoints, benchmark logs, and evaluation reports for 4 distinct ~50M parameter language model architectures trained from scratch on the Amharic Wikipedia corpus using an RL-trained subword tokenizer (3,919 tokens).

📊 Models & Architectures

📁 Repository Structure

├── models/                     # PyTorch architecture implementations
│   ├── transformer_lm.py       # Qwen3.5 3:1 DeltaNet + Attention + MTP
│   ├── hrm_lm.py               # HRM-Text H2L3 recurrence
│   ├── mamba_lm.py             # Mamba selective SSM
│   └── hybrid_lm.py            # Hybrid Mamba + Qwen attention
├── tokenizer/                  # Amharic subword tokenizer
│   ├── vocab.txt
│   └── config.json
├── checkpoints/                # Best model weights for each architecture
│   ├── transformer/best_model.pt
│   ├── hrm/best_model.pt
│   ├── mamba/best_model.pt
│   └── hybrid/best_model.pt
└── analysis/                   # Comparative benchmark results & plots
    ├── report.md
    ├── results_table.tex
    ├── loss_curves.png
    ├── throughput_scaling.png
    └── pareto_frontier.png

🚀 How to Load and Use

import torch
from models import create_model

# Load model architecture
model = create_model("transformer", vocab_size=3919)

# Load checkpoint
checkpoint = torch.load("checkpoints/transformer/best_model.pt", map_location="cpu")
model.load_state_dict(checkpoint["model_state"], strict=False)
model.eval()

# Generate tokens
tokens = torch.tensor([[2, 45, 128, 902]], dtype=torch.long)
with torch.no_grad():
    logits, _ = model(tokens)
    next_token = torch.argmax(logits[:, -1, :], dim=-1)
print("Next token ID:", next_token.item())

📜 Citation & Attribution

If you use these models or comparative benchmarks in your research, please cite:

• Qwen3.5: Alibaba Qwen Team (2025/2026)
• HRM-Text: Sapient Intelligence (Wang et al., 2026)
• Mamba: Gu & Dao (2023)