tsumugi-mrl

Tsumugi-MRL

Pretrained music audio encoder from Tsumugi-MRL. It reads stereo audio at 22,050 Hz and returns 512-dimensional frame features at 25 Hz for downstream music tasks.

Usage

import torch
from tsumugi_mrl import TsumugiMRLModel

model = TsumugiMRLModel.from_pretrained("anime-song/tsumugi-mrl").eval()
audio = torch.randn(1, 2, 22_050 * 4)  # [batch, 2, samples], stereo 22,050 Hz
with torch.no_grad():
    hidden = model(audio)  # [batch, frames, 512], 25 frames per second
    clip = model.mean_pool(hidden)  # [batch, 512]

TsumugiMRLModel lives in the Tsumugi-MRL repository; check it out before loading the weights. Convert mono audio by repeating the channel. The checkpoint carries the Mel normalization statistics used in pretraining, so no separate preprocessing is needed.

Use the frame features or their mean. The model also has an audio_projection head and encode_embedding(), but no pretraining objective trained that head, so its embeddings are not meaningful without fine-tuning.

Architecture

A 128-bin Mel spectrogram (FFT 2,048, hop 441) is computed for each channel, and two adjacent frames are folded together into one 512-dimensional token at 25 Hz. A linear projection feeds an eight-layer Transformer with RoPE, eight heads, a feed-forward width of 2,048, and a final RMSNorm. The encoder has 25.5 M parameters.

Pretraining

The encoder was trained for 625 epochs on 17,037 songs, with 30-second crops and a batch size of 16, by predicting two kinds of targets at masked frames:

• codes from the Mel-RVQ tokenizer (weight 1.0), which describe spectral content;
• codes from the symbolic teacher (weight 0.5), which encodes a transcribed MIDI version of the same song and describes notes, chords, bass, key, beat, and instrumentation.

Both tokenizers emit eight codes per 25 Hz frame. Training used AdamW with a peak learning rate of 5e-4, 8% warmup, and bfloat16 autocast.

Evaluation

Linear probes on frozen features, seed 0, with early stopping on the validation split. Clip tasks use mean-pooled features; GuitarSet and Ballroom are frame-level.

Against the same encoder pretrained on Mel-RVQ codes alone, GuitarSet chord macro-F1 rises from 44.80% to 60.17% and NSynth pitch accuracy from 85.94% to 88.09%. Beat F1 is about two points lower (78.87% for Mel-RVQ alone).

Limitations

• About half of the remaining NSynth pitch errors are octave errors, and pitch accuracy drops above MIDI note 96, a register that is rare in the pretraining music. The Mel front end resolves less than one bin per semitone below middle C.
• The symbolic targets come from automatic transcription, not from verified scores, and inherit its mistakes.
• Pretraining used 30-second crops; longer inputs work but were not seen in training.

License

The weights are released under CC BY-NC 4.0 for non-commercial use.