anlp-a1-transformers-C4

C4 -- RMSNorm instead of LayerNorm

Encoder-decoder transformer written from scratch -- no nn.Transformer, no pretrained weights, tokenizer fitted in-repo -- trained to decipher a binary cipher into English plaintext. One row of a five-configuration ablation in which exactly one architectural choice moves at a time.

Test metrics

• test -- teacher-forced, per window
• test_full -- teacher-forced, per whole line
• gen_test_win -- greedy, per window
• gen_test -- greedy, windowed and stitched
• gen_test_full -- greedy, whole line in one pass

Files

• C4_best.pt -- weights, optimiser state, config and vocabulary sizes, from the best epoch.
• config.json -- the full ExperimentConfig this was trained under.
• summary.json -- the run record: the metrics above plus per-epoch history.
• tokenizer_cipher.json, tokenizer_plain.json -- the fitted unigram vocabularies, when the config uses them. Byte-level configurations have none by construction.

Loading it

The checkpoint is a plain torch.save dict, not a transformers model. The vocabulary sizes travel with it because they are not recoverable from the config -- the tokenizer lands wherever pruning leaves it:

import torch
from huggingface_hub import hf_hub_download

from src.config import get_config
from src.models.base import build_model

path = hf_hub_download("winterdewdev/anlp-a1-transformers-C4", "C4_best.pt")
state = torch.load(path, map_location="cpu", weights_only=False)

cfg = get_config(state["config_name"])
model = build_model(cfg, state["src_vocab_size"], state["tgt_vocab_size"])
model.load_state_dict(state["model"])
model.eval()

Trained with python -m src.train --config C4.