anlp-assignment1-models

ANLP Assignment 1 — C1-C5 Checkpoints

This repository contains the best-validation checkpoints for five custom encoder-decoder Transformer configurations trained for encrypted binary sequence to plaintext reconstruction. The Transformer components were implemented from basic PyTorch operations rather than nn.Transformer or nn.MultiheadAttention.

Repository contents

├── C1/
│   ├── model.pt
│   └── config.json
├── C2/
│   ├── model.pt
│   └── config.json
├── C3/
│   ├── model.pt
│   └── config.json
├── C4/
│   ├── model.pt
│   └── config.json
├── C5/
│   ├── model.pt
│   └── config.json
└── tokenizer/
    └── merge_rules.json

Each config.json contains the resolved hyperparameters, dataset statistics, best checkpoint epoch, and final test metrics. All sequence-level test metrics were computed using greedy decoding from the best-validation checkpoint.

Configurations

C2 is the corrected model in which RoPE is applied independently to Q and K in encoder self-attention, decoder self-attention, and decoder cross-attention.

Shared training setup

• Data split: 4,000 train / 500 validation / 500 test, seed 42
• Global model dimension: 256
• Encoder/decoder layers: 4/4
• Global attention heads: 8
• FFN dimension: 1,024
• Dropout: 0.1
• Batch size: 2
• Epochs: 20
• Optimizer: AdamW
• Learning rate: linear 5% warmup to 3e-4, then cosine decay to 1e-5
• Weight decay: 0.01
• Gradient clipping: 1.0
• Precision: FP16 mixed precision
• Maximum sequence/decode length: 4,096
• Decoding: greedy

Final test results

BLEU and ROUGE apply only to the learned-tokenizer configurations C1-C4.

Tokenizers

C1-C4 use the same frozen binary BPE tokenizer stored in tokenizer/merge_rules.json. It starts from individual binary symbols and uses 1,022 learned variable-length merges. C5 does not use this tokenizer; it uses raw byte IDs with local BLT patch modules.

Loading

These are checkpoints for a custom PyTorch architecture, not Transformers AutoModel packages. Instantiate the matching configuration using the assignment source code, then load the state dictionary:

import torch

checkpoint = torch.load("C2/model.pt", map_location="cpu", weights_only=False)
config = checkpoint["config"]

# Construct the corresponding custom model using config, then:
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

The checkpoint also includes the optimizer state, selected epoch, validation metrics, and resolved configuration.

Limitations

• Results are from one random seed.
• Exact reconstruction is difficult because examples can contain thousands of output positions and one error invalidates the complete sequence.
• C5 loss and throughput are not directly comparable to C1-C4 because it uses a different vocabulary and raw-byte processing.
• These models are intended for coursework and research reproduction, not production or security use.