anlp-a1-transformers-blt

ANLP Assignment 1 — Transformers from Scratch, Architectural Ablation, and BLT

Custom Encoder-Decoder Transformer checkpoints for a controlled 5-config architectural ablation study, trained from scratch (no nn.Transformer / nn.MultiheadAttention) on an encrypted-binary-to-plaintext sequence-to-sequence task. Task: IIIT-H Advanced NLP, Assignment 1.

These are raw PyTorch state_dict checkpoints for a custom architecture. This repo does not include the model source code (submitted separately as part of the assignment zip, not published here). To load a checkpoint, copy src/models/{attention,positional,norm,transformer,blt}.py from the assignment code and:

import torch
from src.models.transformer import Seq2SeqTransformer
from src.models.blt import BLTSeq2SeqTransformer

# C1-C4 (subword BPE, encoder-decoder transformer)
model = Seq2SeqTransformer(
    src_vocab_size=4096, tgt_vocab_size=2048,   # from cipher_bpe_tokenizer.json / plain_bpe_tokenizer.json
    dim=512, num_layers=4, num_heads=16, ff_dim=1024,
    max_seq_len=200, dropout=0.1,
    pe_type="sinusoidal", attention_type="mha", norm_type="layernorm",  # see table below per config
)
model.load_state_dict(torch.load("c1_checkpoint.pt", map_location="cpu"))
model.eval()

# C5 (BLT, token-free byte-level)
blt = BLTSeq2SeqTransformer(
    vocab_size=259, byte_dim=512, patch_dim=512, patch_size=4,
    num_layers=4, num_heads=16, num_kv_heads=16, ff_dim=1024, max_seq_len=200,
    pe_type="sinusoidal", attention_type="mha", norm_type="layernorm", dropout=0.1,
)
blt.load_state_dict(torch.load("c5_checkpoint.pt", map_location="cpu"))
blt.eval()

Task

Map an encrypted binary sequence (0/1 string, 1 byte per plaintext character, verified 1:1 aligned) to its plaintext English sentence. Trained on 5,000 line-aligned pairs (Brown corpus based), windowed into 1024-bit (128-byte/char) non-overlapping chunks per line for training, with train/val/test split (90/5/5, seed 42) done at the line level before windowing to prevent leakage.

Configurations

Each config changes exactly one component from the C1 baseline (dim=512, layers=4, heads=16, ff_dim=1024, dropout=0.1, label_smoothing=0.05, lr=3e-4, batch_size=64, up to 200 epochs with early stopping patience=50):

Both tokenizers (cipher_bpe_tokenizer.json, plain_bpe_tokenizer.json) are included in this repo — from-scratch byte-level / word-level BPE, no tokenizers/sentencepiece library used to train them.

Test set results

Greedy decoding only. Whole-sentence (reassembled) is the number comparable to "Sequence Accuracy" as literally defined in the assignment spec; per-window is a secondary, easier metric (see assignment README for why both are reported).

Whole-sentence (n=250 lines)

Per-window (n=1308 windows)

Training logs

Full training curves and per-run system metrics (peak GPU memory, wall-clock time) are logged on Weights & Biases: velven1206-iiit-hyderabad/ANLP A1 Windowed v3.