universal-translator-ru-en

Universal Translator RU → EN

Model Description

This is a custom autoregressive translator that converts Russian text into English using a Universal Transformer architecture with Mixture of Experts (MoE) and Multi-Head Latent Attention (MLA).

Architecture Highlights

• Type: Universal Transformer with Weight Tying
• Physical Parameters: ~46M in VRAM
• Effective Depth: ~550M (loops the same block 12 times)
• Attention: Multi-Head Latent Attention (DeepSeek-style KV compression)
• Feed-Forward: Mixture of Experts (1 Shared + 8 Routed, Top-K=2)
• Tokenizer: GPT-2 BPE (handles Cyrillic via byte fallback)

Training Data

Trained on the Helsinki-NLP/opus_books parallel corpus (Russian-English book translations).

How to Use This Model

1. Install Requirements

pip install torch tiktoken huggingface_hub

2. Define the Architecture

You must define the model classes (RMSNorm, SwiGLU, MultiHeadLatentAttention, DeepSeekMoE, UniversalBlock, UniversalTranslator) from the training script first.

3. Load and Translate

import torch, json
import tiktoken
from huggingface_hub import hf_hub_download

repo_id = 'antontuzovAI/universal-translator-ru-en'
config_path = hf_hub_download(repo_id=repo_id, filename='config.json')
weights_path = hf_hub_download(repo_id=repo_id, filename='model_weights.pth')

with open(config_path, 'r') as f:
    config = json.load(f)

enc = tiktoken.get_encoding('gpt2')
model = UniversalTranslator(**config)
model.load_state_dict(torch.load(weights_path, map_location='cpu'))
model.eval()

def translate(russian_text):
    prompt = f'RU: {russian_text}\nEN: '
    tokens = enc.encode(prompt, allowed_special={'<|endoftext|>'})
    x = torch.tensor([tokens])
    with torch.no_grad():
        for _ in range(100):
            logits = model(x)
            next_token = torch.argmax(logits[:, -1, :], dim=-1, keepdim=True)
            x = torch.cat([x, next_token], dim=1)
            if next_token.item() == enc.eot_token:
                break
    return enc.decode(x[0].tolist()).split('EN: ')[-1].replace('<|endoftext|>', '').strip()

print(translate('Привет, как дела?'))

Translation Format

The model was trained on autoregressive pairs formatted as:

RU: [Russian text]
EN: [English text]

For best results, keep input sentences concise (under 30 words).