Downloads · 30 days
0
abhirajratna/2024114011-anlp-a1
2024114011-anlp-a1 is a machine learning model from abhirajratna. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
From-scratch encoder-decoder Transformers trained to decrypt a repeating-key XOR cipher (binary string - English plaintext), with a controlled five-configuration ablation over positional encoding, attention, normalisa…
Downloads · 30 days
0
Access
Public
Updated Sep 3, 2026
Repo size
701 MB
Likes
0
Public
Click a slice to open those files.
.pt463 MB · 99%
From the Hugging Face model README
From-scratch encoder-decoder Transformers trained to decrypt a repeating-key XOR cipher (binary string -> English plaintext), with a controlled five-configuration ablation over positional encoding, attention, normalisation and tokenization.
Built with basic PyTorch ops only: no nn.Transformer, no nn.MultiheadAttention, no
F.scaled_dot_product_attention.
| Config | Positional | Attention | Norm | Tokenization |
|---|---|---|---|---|
| C1 base | Sinusoidal | MHA | LayerNorm | BPE |
| C2 | RoPE | MHA | LayerNorm | BPE |
| C3 | Sinusoidal | GQA (8q/2kv) | LayerNorm | BPE |
| C4 | Sinusoidal | MHA | RMSNorm | BPE |
| C5 | Sinusoidal | MHA | LayerNorm | BLT (token-free) |
Each of C2-C5 differs from C1 in exactly one component; all other hyperparameters (depth, width, heads, d_ff, learning rate, batch size, step budget, seed) are identical.
checkpoints/c*.pt - best-validation checkpoint per configurationcheckpoints/entropy_models.pt - the two frozen byte entropy models used by C5's patchertokenizers/ - source (binary) and target (text) BPE tokenizers, vocab 4096results.csv, results.md - the full ablation tableplots/ - metric, cost and length-generalization figuresLoad with the training repository (see the assignment submission):
import torch
from src.config import get_config
ckpt = torch.load("c1_base.pt", map_location="cpu", weights_only=False)
model = get_config("c1").build_model()
model.load_state_dict(ckpt["model"])
| config | changed | params | ms_per_step | peak_mem_mib | bit_accuracy | sequence_accuracy | levenshtein_mean | levenshtein_median | cer | bleu | rouge1 | rouge2 | rougeL | train_minutes | final_val_loss |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| c1_base | none (base) | 18896896 | 45.6597 | 1,736 | 0.9665 | 0.7024 | 0.6577 | 0.0000 | 0.0059 | 95.6662 | 0.9800 | 0.9632 | 0.9800 | 15.9171 | 0.1493 |
| c2_rope | positional encoding | 18896896 | 47.3486 | 1,716 | 0.9698 | 0.7190 | 0.6727 | 0.0000 | 0.0060 | 95.8511 | 0.9808 | 0.9646 | 0.9808 | 16.5054 | 0.1429 |
| c3_gqa | attention mechanism | 16537600 | 42.8520 | 1,705 | 0.9600 | 0.6549 | 0.8101 | 0.0000 | 0.0074 | 94.7472 | 0.9753 | 0.9546 | 0.9753 | 14.9657 | 0.1648 |
| c4_rmsnorm | normalization | 18890752 | 45.1415 | 1,596 | 0.9640 | 0.6838 | 0.7064 | 0.0000 | 0.0062 | 95.1954 | 0.9781 | 0.9593 | 0.9781 | 15.7347 | 0.1560 |
| c5_blt | tokenization | 19504768 | 64.7613 | 1,971 | 0.9975 | 0.9232 | 0.3538 | 0.0000 | 0.0066 | 98.9715 | 0.9949 | 0.9912 | 0.9949 | 22.5258 | 0.0824 |
| config | long_bit_accuracy | long_sequence_accuracy | long_levenshtein_mean | long_levenshtein_median | long_cer | long_mean_pred_len | long_mean_ref_len | long_bleu |
|---|---|---|---|---|---|---|---|---|
| c1_base | 0.6978 | 0.1498 | 70.1147 | 89.0000 | 0.2785 | 144 | 213 | 58.3542 |
| c2_rope | 0.7122 | 0.2495 | 122 | 35.0000 | 0.4807 | 291 | 213 | 58.0447 |
| c3_gqa | 0.6983 | 0.1441 | 68.8251 | 88.0000 | 0.2738 | 146 | 213 | 58.4789 |
| c4_rmsnorm | 0.6940 | 0.1505 | 71.1419 | 90.0000 | 0.2826 | 145 | 213 | 57.3874 |
| c5_blt | 0.6182 | 0.1591 | 95.4036 | 126 | 0.3930 | 120 | 213 | 44.2853 |
C5 originally scored bit_accuracy 0.6549, which is exactly what predicting a space at every
position scores, with 0.0 exact-sequence accuracy. It generated fluent English unrelated to the
source. Teacher-forced val loss of ~1.29 nats/byte is roughly what an unconditional character
LM gets, and 100% of test predictions opened with " the" against 1.5% of references: the model
was language-modelling, not decrypting. Six defects in the BLT pathway, all fixed. None of them
produced a crash, a NaN, or a visibly bad loss curve.
| # | Defect | Fix |
|---|---|---|
| 1 | Global core positioned patches by index, not byte offset. Source patch k and target patch k started a mean 9.9 bytes apart (p90 22, max 73), so cross-attention had no alignment prior and no way to build one -- recovering byte offsets means exactly summing patch lengths across positions | pos_ids threaded through Seq2SeqTransformer; both sides positioned in a shared byte frame. Defaults to None, so C1-C4 are bit-identical |
| 2 | Bytes in target patch 0 could attend to nothing but a learned constant -- the one-patch causality shift lived in the local decoder and pushed patch 0's only global slot off the front | Shift moved to the global decoder's input, so slot 0 cross-attends to the source like every other slot |
| 3 | Source and target were segmented by independent entropy models, so one global slot had to assemble a byte range straddling two or three source patches | blt_tgt_patching="source": the cipher entropy model's cut points carry over. Valid because the cipher is a byte-aligned bijection and the source is available at inference |
| 4 | The pooled patch vector did not contain its own bytes. A single constant pooling query returns a bag summary, not an ordered code | One pooling query per within-patch offset; LocalDecoder expands the patch state back along the same axis |
| 5 | The pooling keys carried only absolute positions, so query j could not address within-patch offset j -- absolute positions shift with wherever a patch starts | Within-patch offset embedding on the pooling keys: the encoder-side mirror of LocalDecoder.in_patch_pos |
| 6 | Patch size traded against reconstruction fidelity and was set too large | blt_max_patch 16 -> 8, blt_target_patch_size 6.0 -> 4.0 |
Defect 4 is the one worth generalising. BLT pools with a single query, and in BLT that is sufficient: it is a decoder-only LM whose local decoder also receives the local encoder's byte-level states through a residual, so exact bytes flow around the patch bottleneck. Ported to seq2seq that residual has no analogue -- every source byte the decoder needs must survive pooling. A bottleneck that destroys information is indistinguishable from an unconverged model: the loss falls smoothly and nothing errors. The permute-the-source probe passed throughout, because the pathway did carry something (patch identity, word shape), just not the bytes. What resolved it was a linear probe for the specific content the pathway must carry, scored against the majority-class baseline -- byte at within-patch offset 1: 0.374 vs a 0.130 baseline before, 0.602 after (0.696 off the global encoder memory).
| variant | val | bit | seq | Levenshtein |
|---|---|---|---|---|
| original (at 20k steps) | 1.2889 | 0.6549 | 0.0000 | 82.66 |
| + byte-coordinate patch positions (1) | 1.4540 | 0.6857 | 0.0000 | 74.37 |
| + global BOS shift (2), source-derived patching (3) | 1.5208 | 0.7459 | 0.0000 | 75.88 |
| + ordered patch codes (4) | 0.8654 | 0.8405 | 0.0000 | 48.33 |
| + within-patch pooling keys (5) | 0.2651 | 0.9728 | 0.0391 | 10.35 |
| + patch geometry 8 / 4.0 (6) | 0.0921 | 0.9959 | 0.6172 | 1.04 |
Val loss rises slightly at the second row while decoded accuracy improves: the two rows use different target segmentations, so per-byte cross-entropy is measuring a different conditioning problem. Decoded metrics are the comparable ones.
| max_patch | mean patch | pooling-key fix | val | bit | seq |
|---|---|---|---|---|---|
| 16 | 5.88 | no | 0.8654 | 0.8405 | 0.0000 |
| 16 | 3.92 | no | 0.3066 | 0.9567 | 0.0078 |
| 8 | 3.93 | no | 0.2737 | 0.9672 | 0.0156 |
| 4 | 2.48 | no | 0.1273 | 0.9899 | 0.2305 |
8, d_local=64 | 3.93 | no | 0.3384 | 0.9618 | 0.0039 |
| 16 | 5.63 | yes | 0.2651 | 0.9728 | 0.0391 |
| 8 | 3.93 | yes | 0.0921 | 0.9959 | 0.6172 |
Larger patches save global compute and cost accuracy. 8 / 4.0 is the knee, and a ~4-byte mean
patch is where the BLT paper operates, so this is not a retreat to near-byte-level segmentation
to buy accuracy. The d_local=64 row tested the hypothesis that the patch code should be
exactly 1:1 with d_model (n_slots x d_local = 8 x 64 = 512); it was worse than the 2x
compressed d_local=128 row, so patch size dominates, not the compression ratio.
best_step is 7000 of 20000 (val 0.0824, against 0.0839 at step 20000). That is a flat
plateau rather than the divergence a low best_step usually signals, but C5 does not use its
full budget while C1-C4 do.bit_accuracy 0.618 vs C1's 0.698),
the one axis where it loses. Entropy patching a 2x-longer source yields a patch count the
global core never trained on.Henna/Jenna, North leet/Northfleet).