anlp-a1-cipher-transformer

Cipher-to-plaintext Transformers (Advanced NLP, Assignment 1)

Encoder-decoder Transformers built from scratch (no nn.Transformer or nn.MultiheadAttention) that decrypt a position-periodic substitution cipher back to English. Five configurations differing from the C1 base by exactly one component each.

Tokenization is hand-written too: a from-scratch BPE over the raw bit string on the cipher side and over characters on the English side (C1-C4), and byte-level input with entropy-chosen patches for C5. data/bpe_*.json and data/entropy_ngram.npz are part of the checkpoint -- the weights are unusable without them.

Measured on the whole 1000-line test split (data/splits/test.txt) under greedy decoding: each line is tiled into windows, decoded, reassembled, and scored as a whole line. C5's BLEU and ROUGE-L read --, not 0 -- the assignment restricts them to tokenized models and C5 is byte-level.

Cost

Files

c1/best.ckpt .. c5/best.ckpt   one checkpoint per configuration
data/bpe_plain.json            plaintext BPE vocabulary (C1-C4)
data/bpe_cipher.json           cipher BPE vocabulary (C1-C4)
data/entropy_ngram.npz         order-1 entropy estimator for C5's patcher
data/splits/                   the train/test partition behind every number

hf download aadith-warrier/anlp-a1-cipher-transformer --local-dir anlp-a1

Loading a checkpoint

import torch
from omegaconf import OmegaConf
from models.transformer import build_model

ck = torch.load('c1/best.ckpt', map_location='cpu', weights_only=False)
cfg = OmegaConf.create(ck['hyper_parameters'])   # the run's own config
model = build_model(cfg, 259, 2048)              # 259 for byte targets (C5)
model.load_state_dict({k[6:]: v for k, v in ck['state_dict'].items()
                       if k.startswith('model.')})

build_model is not in this repo: the snippet needs the assignment's src/ on PYTHONPATH. data/bpe_plain.json is the exact BPE vocabulary these checkpoints were trained against, and data/splits/ is the train/test partition, so the numbers above are reproducible.