anlp-a1-assignment

ANLP Assignment 1 — Encoder–Decoder Ablations (C1–C5)

This hub entry hosts checkpoints from a controlled ablation: map encrypted binary sequences (brown_cipher.txt) to English plaintext (brown_plain.txt). Configuration C1 is the base model; C2–C5 each change exactly one component.

---

Model variants

Shared hyperparameters

Tokenization

• C1–C4: from-scratch BPE, vocab 1024 per side. Cipher: pack 8 bits → byte, then learn merges. Plain: character BPE. Artifacts: cipher_bpe.json, plain_bpe.json.
• C5: no BPE. Packed bytes + entropy patches (len 1–8, mean ≈ 4.2), hashed 2/3/4-gram features. Artifacts: entropy_ngram.json, entropy_patcher.json.

---

Intended use

• Primary: course ablation / reproducibility for ANLP Assignment 1.
• Task: greedy decode ciphertext chunks → plaintext chunks.
• Not intended for: production MT, open-domain generation, or claiming a general “decrypt any cipher” system. The mapping is specific to this homework dataset.

---

How to load weights

Checkpoints are raw PyTorch state dicts (pytorch_model.bin) plus config.json. They are not Hugging Face transformers.AutoModel checkpoints—load with the assignment code.

import json
import torch
from pathlib import Path

# after cloning / downloading this repo's `src/` onto PYTHONPATH
from models.transformer import Seq2SeqTransformer
from models.blt import BLTSeq2Seq
from tokenizer import BPETokenizer, PAD

hf_dir = Path("path/to/downloaded/hf")  # folder with config.json + pytorch_model.bin
cfg = json.loads((hf_dir / "config.json").read_text())

if cfg.get("blt"):
    model = BLTSeq2Seq(
        d_model=cfg["d_model"],
        n_heads=cfg["n_heads"],
        n_kv_heads=cfg["n_kv_heads"],
        n_encoder_layers=cfg["n_encoder_layers"],
        n_decoder_layers=cfg["n_decoder_layers"],
        d_ff=cfg["d_ff"],
        dropout=cfg["dropout"],
        patch_size=cfg["patch_size"],
        n_local_layers=cfg["n_local_layers"],
        n_local_heads=cfg["n_local_heads"],
        pad_id=PAD,
    )
else:
    model = Seq2SeqTransformer(
        src_vocab=cfg["src_vocab"],
        tgt_vocab=cfg["tgt_vocab"],
        d_model=cfg["d_model"],
        n_heads=cfg["n_heads"],
        n_kv_heads=cfg["n_kv_heads"],
        n_encoder_layers=cfg["n_encoder_layers"],
        n_decoder_layers=cfg["n_decoder_layers"],
        d_ff=cfg["d_ff"],
        dropout=cfg["dropout"],
        pos_encoding=cfg["pos_encoding"],
        attn_kind=cfg["attn_kind"],
        norm_kind=cfg["norm_kind"],
        pad_id=PAD,
    )

state = torch.load(hf_dir / "pytorch_model.bin", map_location="cpu")
model.load_state_dict(state)
model.eval()

---

Training data

• Source: assignment corpus brown_cipher.txt / brown_plain.txt (5000 line pairs).
• Split: document-level 80/10/10, seed 42, then aligned 256-character chunks (~11.5k / 1.4k / 1.4k train/val/test).
• Property: len(cipher)/8 == len(plain) so packed cipher byte i aligns with plaintext character i.

---

Evaluation

Greedy decoding only. Test metrics (1389 chunks):

*BLEU/ROUGE are secondary for token-free C5; greedy C5 BLEU ≈ 89.3 if computed for reference.

Ranking: C2 > C1 ≈ C4 > C3 > C5.

Note on C5: bit accuracy stays relatively high while sequence (exact-match) accuracy is lower because errors are mostly scattered typos on long chunks (mean Lev ≈ 3.3). Parallel patch decoding and entropy cuts on predicted bytes amplify that gap.

---

C1–C4

C5

---

Limitations

• Single seed (42).
• C5 is not capacity-matched to C1 (extra byte/n-gram parameters).
• Results are specific to this index-aligned local cipher, not general MT.
• Not an transformers Auto* model; requires the course src/ code to run.
• Do not treat published weights as a cryptanalysis tool beyond this homework setup.

---

References

• Vaswani et al., Attention Is All You Need, NeurIPS 2017.
• Sennrich et al., Neural Machine Translation of Rare Words with Subword Units, ACL 2016.
• Su et al., RoFormer (RoPE), 2021.
• Ainslie et al., GQA, 2023.
• Zhang & Sennrich, RMSNorm, NeurIPS 2019.
• Pagnoni et al., Byte Latent Transformer, ACL 2025. https://aclanthology.org/2025.acl-long.453