notes-you-cant-delete

Notes You Can't Delete — model checkpoints

Small from-scratch symbolic-music transformers from the study Notes You Can't Delete: take a musical idea out of everything a model is trained on, and it rebuilds the idea anyway.

Each model is a ~25–26M-parameter decoder-only GPT (nanoGPT shape) trained on REMI-tokenized MIDI. Every checkpoint is a matched control or edited twin — the whole point of the study is comparing pairs that differ in exactly one thing.

Full write-up, audio, and the numbers: https://notes-you-cant-delete.vercel.app

Run one

pip install torch numpy miditok symusic huggingface_hub
python generate_example.py --model g_metal_s1     # writes out.mid

generate_example.py and model.py (the ~120-line GPT) are in this repo. Each checkpoint is checkpoints/<name>/ckpt.pt (a state dict + config) with its config.json; tokenizers are in tokenizers/.

The checkpoints

Metal (guitar & full band)

Tonal (classical & jazz)

The one result

Delete a concept from the training data — a note, a chord, a whole genre — and a model that never sees it once still expects it, reconstructed from the grammar around the hole. The blue note comes back at 97–98%, the metal tritone at 99%, the dominant seventh at ~90%, black metal fully. The only place deletion finally bites is a genuinely novel trait (djent's sub-bass register) that nothing else in the data implies.

Corpora: Weimar Jazz Database, MAESTRO v3, and a public Kaggle metal-MIDI dump (PDDL). Models are small-scale and from-scratch; caveats are stated throughout the site.