anlp-a1-transformers

RaunakSeksaria/anlp-a1-transformers

anlp-a1-transformers is a machine learning model from RaunakSeksaria. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch.

Five encoder-decoder transformers trained from scratch to map an XOR-encrypted binary sequence to English plaintext. Each configuration differs from the base by exactly one architectural component.

Downloads · 30 days

Access

Public

Updated Sep 2, 2026

Repo size

816 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.pt509 MB · 100%

At a glance

Library: pytorch
Access: Public
Created: Sep 2, 2026
Updated: Sep 2, 2026
SHA: 73e92cf0

Model card

From the Hugging Face model README

ANLP Assignment 1 - encoder-decoder transformer ablation

Five encoder-decoder transformers trained from scratch to map an XOR-encrypted binary sequence to English plaintext. Each configuration differs from the base by exactly one architectural component.

Config	Change from base	Checkpoint
C1	base: sinusoidal positions, multi-head attention, LayerNorm, subword	`c1_best.pt`
C2	RoPE instead of sinusoidal positions	`c2_best.pt`
C3	grouped-query attention instead of multi-head	`c3_best.pt`
C4	RMSNorm instead of LayerNorm	`c4_best.pt`
C5	token-free byte latent transformer instead of subword	`c5_best.pt`

Test results (greedy decoding)

Config	Bit acc	Seq acc	Levenshtein	BLEU	ROUGE-L
C1	0.8960	0.3203	2.9023	0.8645	0.9305
C2	0.9482	0.5316	1.5340	0.9215	0.9598
C3	0.8701	0.2129	4.5672	0.8120	0.9043
C4	0.8972	0.3164	2.9723	0.8609	0.9283
C5	0.9987	0.6574	0.6363

Loading a checkpoint

import torch
state = torch.load('c1_best.pt', map_location='cpu')
state['config']       # the hyperparameters used
state['model']        # the weights

Checkpoints also carry optimizer and scheduler state so training can resume. Model code is in the accompanying submission, not on the Hub.

Library: pytorch
Created: Sep 2, 2026
Updated: Sep 2, 2026