2025801013_anlp_assignment1

Cipher-to-Plaintext Transformers — Architectural Ablation

Encoder–decoder transformers built from fundamental PyTorch operations (no nn.Transformer, no nn.MultiheadAttention) for ANLP Assignment 1.

The task: recover English plaintext from a byte-wise repeating-key XOR cipher with an 8-byte key, one byte (8 bits) per plaintext character. The key itself is a property of the course dataset and is not published here.

Configurations

Each differs from the C1 baseline by exactly one component.

Shared hyperparameters: d_model 256, 8 heads, 6 layers, d_ff 1024, dropout 0.1, AdamW at 3e-4, batch 32, 20 epochs, 64-character chunks.

Results

Greedy decoding on a held-out test split of 4,371 chunks.

Measured noise floor (C1, three seeds): ±0.0175pp character accuracy. Differences smaller than that are not interpreted.

Files

• C1_BitPairTokenizer_gelu_best.pt (45.9 MB)
• C1_BytePairTokenizer_gelu_best.pt (45.9 MB)
• C1_BytePairTokenizer_relu_best.pt (45.9 MB)
• C2_BitPairTokenizer_gelu_best.pt (45.4 MB)
• C2_BytePairTokenizer_gelu_best.pt (45.4 MB)
• C2_BytePairTokenizer_relu_best.pt (45.4 MB)
• C3_BitPairTokenizer_gelu_best.pt (38.8 MB)
• C3_BytePairTokenizer_gelu_best.pt (38.8 MB)
• C3_BytePairTokenizer_relu_best.pt (38.8 MB)
• C4_BitPairTokenizer_gelu_best.pt (45.8 MB)
• C4_BytePairTokenizer_gelu_best.pt (45.8 MB)
• C4_BytePairTokenizer_relu_best.pt (45.8 MB)
• C5E_ByteTokenizer_gelu_best.pt (60.3 MB)
• C5_ByteTokenizer_gelu_best.pt (59.3 MB)
• C5_ByteTokenizer_relu_best.pt (59.3 MB)
• histories/C1_BitPairTokenizer_gelu_history.json (0.0 MB)
• histories/C1_BytePairTokenizer_gelu_history.json (0.0 MB)
• histories/C1_BytePairTokenizer_relu_history.json (0.0 MB)
• histories/C2_BitPairTokenizer_gelu_history.json (0.0 MB)
• histories/C2_BytePairTokenizer_gelu_history.json (0.0 MB)
• histories/C2_BytePairTokenizer_relu_history.json (0.0 MB)
• histories/C3_BitPairTokenizer_gelu_history.json (0.0 MB)
• histories/C3_BytePairTokenizer_gelu_history.json (0.0 MB)
• histories/C3_BytePairTokenizer_relu_history.json (0.0 MB)
• histories/C4_BitPairTokenizer_gelu_history.json (0.0 MB)
• histories/C4_BytePairTokenizer_gelu_history.json (0.0 MB)
• histories/C4_BytePairTokenizer_relu_history.json (0.0 MB)
• histories/C5E_ByteTokenizer_gelu_history.json (0.0 MB)
• histories/C5_ByteTokenizer_gelu_history.json (0.0 MB)
• histories/C5_ByteTokenizer_relu_history.json (0.0 MB)
• figures/attention_C1.png (0.0 MB)
• figures/attention_C2.png (0.0 MB)
• figures/attention_C3.png (0.0 MB)
• figures/attention_C4.png (0.0 MB)
• figures/attention_C5.png (0.0 MB)
• figures/bars_memory.png (0.1 MB)
• figures/bars_seq_acc.png (0.1 MB)
• figures/bars_speed.png (0.1 MB)
• figures/bitbpe_attention_C1.png (0.1 MB)
• figures/bitbpe_attention_C2.png (0.0 MB)
• figures/bitbpe_attention_C3.png (0.0 MB)
• figures/bitbpe_attention_C4.png (0.1 MB)
• figures/bitbpe_bars_memory.png (0.0 MB)
• figures/bitbpe_bars_seq_acc.png (0.0 MB)
• figures/bitbpe_bars_speed.png (0.0 MB)
• figures/bitbpe_curves_val_acc.png (0.1 MB)
• figures/bitbpe_curves_val_loss.png (0.1 MB)
• figures/c5e_attention_C5E.png (0.0 MB)
• figures/c5e_bars_memory.png (0.0 MB)
• figures/c5e_bars_seq_acc.png (0.0 MB)
• figures/c5e_bars_speed.png (0.0 MB)
• figures/c5e_curves_val_acc.png (0.1 MB)
• figures/c5e_curves_val_loss.png (0.0 MB)
• figures/curves_val_acc.png (0.1 MB)
• figures/curves_val_loss.png (0.2 MB)
• bpe_vocab1024.json (0.0 MB)
• results.json (0.0 MB)
• heldout_result.json (0.0 MB)

Loading a checkpoint

These are raw state_dict files for an architecture implemented from scratch, so they need this project's model code:

from train import build, TOKENIZERS
import torch

tokenizer = TOKENIZERS["bpe"]()          # "byte" for C5
model = build("C1", tokenizer, activation="gelu")
model.load_state_dict(torch.load("C1_BytePairTokenizer_gelu_best.pt"))
model.eval()

bpe_vocab1024.json is the trained tokenizer and is required for C1–C4; C5 uses raw bytes and needs no vocabulary.

Notes

• Reported numbers use greedy decoding, as the assignment specifies. Accuracy logged during training is teacher-forced and is not comparable.
• BLEU and ROUGE are reported for the tokenized configurations only; they saturate near 0.99 on this task and do not discriminate between configurations.
• A held-out combination experiment shows the model memorises a phase-independent byte→character table rather than inferring the cipher. Withholding 20 of the 424 (character, phase) combinations drops accuracy on them from 99.96% to 23.8% — and every combination whose key byte is unique across the key scores exactly 0%. The residual comes entirely from one key byte that repeats at two phases, so those combinations produce cipher bytes the model had already seen elsewhere. No structural generalisation occurs.