anlp-a1

ANLP Assignment 1 — Transformers from Scratch, Architectural Variants & BLT

This repo hosts the model checkpoints for a from-scratch (no nn.Transformer / nn.MultiheadAttention) sequence-to-sequence Transformer trained to decode a binary-cipher → plaintext mapping derived from the Brown corpus, across a 5-way architectural ablation (C1–C5).

Code: see the accompanying GitHub/submission repo (src/models/, src/train.py, src/dataset.py, src/utils.py).

Files

Each configuration changes exactly one component relative to the C1 baseline.

Architecture / training hyperparameters

Trained on an NVIDIA RTX 3060 GPU.

Dataset

Line-aligned binary-cipher → plaintext pairs derived from the Brown corpus. Sentences with cipher length ≤ 4424 bits (~553 chars, ≈ median sentence length) were kept (2500 total), split 80:10:10 into train/val/test (n_train=2000, n_val=250, n_test=250).

Results (test set, greedy decoding)

BLEU/ROUGE are not computed for C5, per the assignment spec ("for tokenized models only").

You may access the full train trace at https://wandb.ai/aviral-gupta-iiit-hyderabad/anlp-a1

Takeaways: RoPE (C2) gives by far the largest quality improvement over sinusoidal PE (seq acc 0.184 → 0.928). RMSNorm (C4) is roughly on par with LayerNorm. GQA (C3) trades a small amount of accuracy for reduced K/V capacity. The token-free BLT model (C5) trains ~4–5x faster and uses ~5.6x less peak GPU memory than the tokenized configs, at the cost of accuracy — mainly due to difficulty terminating generation cleanly rather than poor content reconstruction.

Loading a checkpoint

Checkpoints are plain torch.save(state_dict, ...) files matching the model classes defined in src/train.py / src/models/. Example:

import torch
from huggingface_hub import hf_hub_download

ckpt_path = hf_hub_download(repo_id="avi1o1/anlp-a1", filename="C2.pt")
state_dict = torch.load(ckpt_path, map_location="cpu")

# Rebuild the matching model architecture from src/train.py with the
# hyperparameters above (config="C2"), then:
# model.load_state_dict(state_dict)

Author: Aviral Gupta (2023111023)