vanilla-transformer-georgian-english

Vanilla Transformer — Georgian ⇄ English Machine Translation from Scratch

A bidirectional Georgian ↔ English neural machine translation model, built by implementing the "Attention Is All You Need" (Vaswani et al., 2017) Transformer architecture from first principles in PyTorch — no torch.nn.TransformerEncoder, no pretrained weights, no HuggingFace model classes.

Trained entirely from random initialization for educational purposes: a faithful reproduction of the original 2017 Transformer, with two small, deliberate modern departures (GELU activation, pre-LayerNorm) for training stability.

Model Details

• Architecture: Encoder-decoder Transformer, hand-implemented (Linear, Embedding, LayerNorm, multi-head self-/cross-attention, sinusoidal positional encoding)
• Modifications from the original paper: GELU instead of ReLU; Pre-LayerNorm instead of Post-LayerNorm (the modern default since GPT-2, for more stable training)
• Directionality: A single model handles both directions via a source-side tag (2en / 2ka) and shared encoder-decoder parameters — not two separate models
• Training hardware: Single NVIDIA L4 GPU, no pretraining
• Loss: Label-smoothed cross-entropy — final training loss ~4.5 (label-smoothed), equivalent to ~3.7 without smoothing

Dataset & Tokenization

• Corpus: Darsala/english_georgian_corpora
• Tokenizer: Joint SentencePiece BPE tokenizer (~32,000 vocabulary), shared across both languages, trained on the parallel corpus

Evaluation

Scored with COMET (wmt22-comet-da) on a held-out test split, greedy decoding:

Scores in this range reflect moderate translation quality — reliable on common, everyday sentence structures, with an expected drop-off on rare vocabulary and specialized domains, given the model was trained entirely from scratch with no pretraining.

Usage

First, install the model architecture package directly from GitHub:

pip install git+[https://github.com/N1k0l0z/Vanilla_Transformer.git](https://github.com/N1k0l0z/Vanilla_Transformer.git)

import torch
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

# 1. Import model architecture and configuration
from Scripts.Transformer import TranslationModel
from Scripts.ModelConfig import Config

# 2. Download assets from Hugging Face Hub
REPO_ID = "N1k0l0z/vanilla-transformer-georgian-english"

model_path = hf_hub_download(repo_id=REPO_ID, filename="model.pt")
tokenizer_path = hf_hub_download(repo_id=REPO_ID, filename="tokenizer.json")

# 3. Load tokenizer and PyTorch checkpoint
tokenizer = Tokenizer.from_file(tokenizer_path)
checkpoint = torch.load(model_path, map_location="cpu", weights_only=False)

# 4. Initialize model and load trained weights
model = TranslationModel(checkpoint["config"])
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

# 5. Translate!
print(model.translate("ვინ ხარ?", tokenizer, direction="2en"))  
# Output: "Who are you?"

print(model.translate("Where is your house?", tokenizer, direction="2ka"))
# Output: "სად არის შენი სახლი?" 

Examples

Georgian → English

English → Georgian

Limitations

• No pretraining — quality is bounded by what a from-scratch model can learn from the training corpus alone
• Weaker on rare, technical, or specialized vocabulary due to limited coverage in training data
• Best suited for everyday, conversational language rather than domain-specific translation

Citation

If referencing the original architecture:

Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS.