ANLP_A1_Assignment_C1-C5

Transformers from Scratch & Byte Latent Transformers

<p align="center"> <b>Advanced NLP — Assignment 1</b><br> Transformers from Scratch · RoPE · GQA · RMSNorm · BLT </p>

---

👤 Author

Shourya Pillai · MS Research, CSE · IIIT Hyderabad
Roll No. 2026701047 · [email protected]

---

🚀 Overview

This project explores Transformer architectures for encrypted-text reconstruction, comparing standard subword Transformers against a Byte Latent Transformer (BLT) operating directly on raw bytes.

All Transformer components were implemented from scratch in PyTorch, without using nn.Transformer or nn.MultiheadAttention.

The experiment consists of five controlled configurations:

---

📊 Results

Final Test Performance

C5 achieves 99.988% bit-level accuracy and 98.599% exact sequence accuracy.

BLEU and ROUGE are not applicable to C5 because it operates directly on bytes rather than subword tokens.

---

🧠 What Changed?

C1 — Baseline

Sinusoidal absolute positional encoding + Multi-Head Attention + LayerNorm + custom BPE.

C2 — RoPE

Replaces absolute positional encoding with Rotary Positional Encoding.

→ Best-performing subword Transformer.

C3 — GQA

Replaces standard MHA with Grouped-Query Attention, reducing the number of key/value heads while retaining multiple query heads.

C4 — RMSNorm

Replaces LayerNorm with RMSNorm while keeping the remaining architecture unchanged.

C5 — BLT

Moves from subword tokens to raw bytes, using a Byte Latent Transformer architecture with local byte processing and global Transformer representations.

→ Near-perfect reconstruction.

---

📈 Training

Training Loss

Validation Loss

Learning Rate

Peak GPU Memory

---

🔑 Key Takeaways

1. RoPE improves the standard subword Transformer.

C2 substantially outperforms the baseline C1 across bit accuracy, sequence accuracy, Levenshtein distance, BLEU, and ROUGE.

2. GQA is competitive with the baseline but does not improve reconstruction quality in this experiment.

3. RMSNorm does not outperform LayerNorm for this task.

4. Byte-level modeling is extremely effective for this reconstruction problem.

C5 dramatically outperforms all subword-based configurations, achieving near-perfect reconstruction.

---

📦 Repository Contents

C1/                 C1 checkpoint
C2/                 C2 checkpoint
C3/                 C3 checkpoint
C4/                 C4 checkpoint
C5/                 C5 BLT checkpoint

tokenizers/
├── tokenizer.json
└── cipher_tokenizer.json

metrics/
└── evaluation_results.csv

plots/
├── C1_C5_Train_Loss.png
├── C1_C5_Validation_Loss.png
├── C1_C5_Learning_Rate.png
└── Peak_GPU_Memory.png