matpac_audioset_finetune_encoder

MATPAC++ (AudioSet fine-tuned encoder)

MATPAC (Masked latent Prediction And Classification) is a self-supervised audio and music encoder. This repo hosts the MATPAC++ encoder, self-supervised then fine-tuned on AudioSet, encoder only (matpac_plus_as_48_1_map_enconly.pt), a transformers-native, trust_remote_code port of the official inference code (aurianworld/matpac, folder inference_matpac).

• 📝 MATPAC: Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning
• 📝 MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning

If you want ready-made AudioSet class predictions instead of embeddings, use matpac_audioset_finetune_classifier (same fine-tuning, plus the 527-way classification head).

The MATPAC family

Usage

from transformers import AutoModel, Wav2Vec2FeatureExtractor
import torch
import torchaudio

model = AutoModel.from_pretrained("auriankelen/matpac_audioset_finetune_encoder", trust_remote_code=True)
processor = Wav2Vec2FeatureExtractor.from_pretrained("auriankelen/matpac_audioset_finetune_encoder", trust_remote_code=True)

waveform, sr = torchaudio.load("my_file.wav")
waveform = waveform.mean(dim=0)  # mono

resample_rate = processor.sampling_rate
if resample_rate != sr:
    waveform = torchaudio.transforms.Resample(sr, resample_rate)(waveform)

inputs = processor(waveform, sampling_rate=resample_rate, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

# outputs.last_hidden_state: (batch, 3840) -- mean-pooled over time by default
# outputs.hidden_states: tuple of 12 tensors, one per encoder layer, each (batch, 3840)
all_layer_hidden_states = torch.stack(outputs.hidden_states)
print(all_layer_hidden_states.shape)  # [12, batch, 3840]

Keeping the time dimension / faster inference

# Keep the time axis instead of mean-pooling it
outputs = model(**inputs, pull_time_dimension=False)
# outputs.last_hidden_state: (batch, time, 3840)

# Vectorized inference: faster on large batches, adds some padding so it is
# slightly less precise than the default ("precise", used for the paper's results)
outputs = model(**inputs, inference_type="fast")

Requirements

This model needs a couple of packages beyond transformers to run its trust_remote_code files: torchaudio, einops, and timm==0.4.12 (the exact timm version the checkpoints were exported with the internal naming of the ViT block submodules must match for the weights to load correctly).

Citation

@inproceedings{quelennec2025matpac,
  title={Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning},
  author={Quelennec, Aurian and Chouteau, Pierre and Peeters, Geoffroy and Essid, Slim},
  booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2025},
  url={https://ieeexplore.ieee.org/document/10887666},
  doi={10.1109/ICASSP49660.2025.10887666}
}

@article{quelennec2025matpacenhancedmaskedlatent,
  title={MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning},
  author={Aurian Quelennec and Pierre Chouteau and Geoffroy Peeters and Slim Essid},
  journal={arXiv preprint arXiv:2508.12709},
  year={2025},
  url={https://arxiv.org/abs/2508.12709}
}

Credits

• aurianworld/matpac for the original training and inference code.
• Fairseq for the training framework.
• M2D for the base of the inference code.
• DINO for the classification head.