anlp-a1-checkpoints

ANLP Assignment 1: Transformers from Scratch, Ablations, and BLT

Encoder-decoder transformer built from basic PyTorch ops that learns to map encrypted binary sequences (brown_cipher.txt) to plaintext English (brown_plain.txt), plus a five-way ablation study over positional encodings (Sinusoidal vs RoPE), attention (MHA vs GQA), normalization (LayerNorm vs RMSNorm), and tokenization (subword vs token-free BLT).

C1–C4 use independently learned, from-scratch BPE tokenizers: 512 source tokens over ciphertext bytes and 1,024 target tokens over plaintext. Merged tokens retain their underlying byte/character offsets for positional encoding. C5 groups ciphertext bits into bytes and uses train-only Markov next-byte entropy to create dynamic patches.

Setup

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
wandb login          # for training-run logging
huggingface-cli login  # for checkpoint uploads

Place the dataset files in the repo root (they are git-ignored and never committed):

ANLP-A1/
├── brown_cipher.txt
├── brown_plain.txt
└── ...

(Or edit cipher_path / plain_path in the configs.)

Running

Train one configuration:

python -m src.train --config configs/c1_base.yaml

Train all five ablation configs:

bash scripts/run_all.sh

Ablation configurations

All other hyperparameters are identical across configs.

Repository layout

src/
├── models/
│   ├── attention.py    # Scaled dot-product attention, MHA, GQA
│   ├── positional.py   # Sinusoidal PE, RoPE
│   ├── norm.py         # LayerNorm, RMSNorm (from scratch)
│   ├── transformer.py  # Encoder/decoder layers, Seq2SeqTransformer
│   └── blt.py          # Local encoder/decoder + BLT model (C5)
├── dataset.py          # Tokenized (C1-C4) and token-free (C5) loaders
├── train.py            # Config-driven training loop with WandB
└── utils.py            # Greedy decoding, metrics, plots
configs/                # One YAML per ablation config
outputs/                # Plots, logs, metrics (checkpoints git-ignored)
scripts/run_all.sh      # Run C1-C5 sequentially

Links

• WandB project: https://wandb.ai/yajatlakhanpal31-iiit-hyderabad/anlp-a1/workspace?nw=nwuseryajatlakhanpal31

• Hugging Face checkpoints: https://huggingface.co/Yajat31/anlp-a1-checkpoints

Evaluation

All reported numbers use greedy decoding. Metrics: bit-level accuracy, sequence accuracy, Levenshtein distance, and BLEU/ROUGE (tokenized models only). See src/utils.py.

Final test BLEU: C1 84.93, C2 98.68, C3 63.66, and C4 83.27. C5 reaches 0.726 bit accuracy while using the lowest peak GPU memory (1,333 MB).