anlp-a1-transformers

ANLP Assignment 1 — Custom Transformers & Byte Latent Transformers

Encoder–decoder Transformers built from scratch in PyTorch (no nn.Transformer, no nn.MultiheadAttention) for the task of mapping encrypted binary sequences to plaintext, plus a controlled five-way architectural ablation.

Configurations

How C5 (BLT) works

• Bytes, not the text of bytes. The ciphertext is a binary sequence stored as ASCII 0/1 characters. Every 8 bits are packed into one byte value 0-255, so a 256-character cipher segment is 32 bytes, not 256.
• Entropy-based dynamic patching. Patches are variable length: a new patch opens where H(x_t | x_{t-2}, x_{t-1}) > theta. The entropy comes from a lightweight order-2 byte n-gram model fit on the training split only (entropy_model.json, also stored as buffers inside best.pt); no separate neural LM is trained. theta is calibrated to a target mean patch length, giving ~10 variable-length patches per 34-byte sequence.
• A standard learned 256-entry byte embedding represents the byte values.

Repository layout

C1/best.pt            checkpoint (state_dict + args + config)
C1/config.json        exact CLI arguments used
C1/metrics.json       test metrics, timings, peak memory
C1/tokenizer_*.json   the from-scratch byte-level BPE vocab + merges
...                   idem for C2..C5
C5/entropy_model.json the n-gram entropy model defining patch boundaries

Loading a checkpoint

import torch
from src.configs import get_config, build_model
import argparse

ck   = torch.load("C1/best.pt", map_location="cpu", weights_only=False)
cfg  = get_config(ck["config"]["name"])
args = argparse.Namespace(**ck["args"])
model = build_model(cfg, args, ck["src_vocab_size"], ck["tgt_vocab_size"])
model.load_state_dict(ck["model_state"])
model.eval()

Training/evaluation code: see the accompanying assignment submission (src/train.py, src/run_all.py).