Downloads · 30 days
0
anlp-a2-part1-moe_active_matched
neemon/anlp-a2-part1-moe_active_matched
anlp-a2-part1-moe_active_matched is a translation model from neemon. Use it when you need text moved from one language to another. It is set up for pytorch. The card lists the license as mit.
Decoder-only transformer with Mixture-of-Experts, 4 experts, top-2 routing, active parameters matched to the dense baseline.
Access
Public
Updated Oct 1, 2026
Repo size
701 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt234 MB · 99%
At a glance
- Task
- Translation
- Library
- pytorch
- License
- mit
- Access
- Public
- Created
- Sep 14, 2026
- Updated
- Oct 1, 2026
- SHA
- 28d575ec
Try a prompt
Model card
From the Hugging Face model README
ANLP Assignment 2 (part1): moe_active_matched
Decoder-only transformer with Mixture-of-Experts, 4 experts, top-2 routing, active parameters matched to the dense baseline.
Trained from scratch for IIIT-H Advanced NLP, Monsoon 2026. Architecture, training code and evaluation live in the assignment repository. This repository holds the checkpoint and the tokenizer it was trained with, which every model in the assignment shares.
Architecture
| Setting | Value |
|---|---|
d_model | 512 |
n_layers | 8 |
n_heads | 8 |
n_kv_heads | 8 |
n_ctx | 256 |
vocab_size | 32000 |
ffn | moe_active_matched |
d_ff | 1024 |
n_routed_experts | 4 |
n_shared_experts | 0 |
top_k | 2 |
norm | rmsnorm |
Parameters
| Count | Value |
|---|---|
| Total | 58,401,280 |
| Active per token | 41,599,488 |
| Feed-forward (total / active) | 33,603,584 / 16,801,792 |
Test set results
| Metric | Value |
|---|---|
| Perplexity (both directions) | 8.81 |
| BLEU vi->en | 35.06 |
| BLEU ja->en | 25.17 |
| BLEU mean | 30.12 |
Training
- Scored target tokens: 33,393,292
- Best validation loss: 1.9321
- Best validation perplexity: 6.90
Loading
import torch
payload = torch.load("model.pt", map_location="cpu", weights_only=False)
state_dict, config = payload["model"], payload["config"]
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
- Task
- Translation
- Library
- pytorch
- License
- mit
- Languages
- en, vi, ja
- Created
- Sep 14, 2026
- Updated
- Oct 1, 2026