leitmotif

Leitmotif — checkpoints

Weights for Leitmotif: reading a song two ways, as a graph of its own structure and through the words people use to describe it. One-second segments become graph nodes, edges join segments that follow each other or sound alike, a GNN reads that structure and BERT reads the caption.

Try it: 404mahdi/leitmotif Space

Files

Usage

These load through the project's own inference wrapper, which expects them in checkpoints/:

from huggingface_hub import hf_hub_download
from src.inference import Leitmotif

for name in ["stage4_contrastive_pretrained.pt", "stage4_contrastive_pretrained_args.json",
             "stage4_contrastive_pretrained_node_stats.pt", "stage4_contrastive_pretrained_index.pt"]:
    hf_hub_download("404mahdi/leitmotif", name, local_dir="checkpoints")

lm = Leitmotif()
lm.search("a sad piano ballad with strings")   # closest MusicCaps clips
lm.tags("song.wav", "a mellow piano ballad")   # predicted context tags
lm.explain("song.wav", "a mellow piano ballad")  # words x segments attention

Results

Retrieval over 2,555 MusicCaps test clips, chance R@10 = 0.004:

Tagging on the same clips, captions with tag words masked, 50 tags:

Limitations, stated plainly

On aggregate tagging metrics the audio graph does not beat text alone — the cross-attention model shipped here scores below the BERT-only baseline. It is published because the per-tag picture is the actual finding: audio helps where a masked caption goes quiet, on who is singing and which instruments play, and costs accuracy on tags the remaining words still describe. The full ablation is in the paper.

Trained on one RTX 5060 Laptop GPU (8 GB) on 10-second clips at 16 kHz. Retrieval quality is roughly 20× chance, not production search. English captions only, and it inherits MusicCaps' genre and cultural skew.

Citation

Solo project by Mahdi Hasan for CSE425 (Neural Networks), BRAC University. Code MIT licensed; MusicCaps is CC BY-SA 4.0 and keeps its own terms.