gramt-binaural-frame

Model Card for Model ID

GRAM (General Purpose Audio Representation Model) is trained on AudioSet with newly proposed naturalistic training methadology. GRAMs utilize MWMAE (Multi-window multi-head attention), and RIR augmentations to achieve state-of-the-art results on downstream tasks such as FSD50K, ESC50, VL even when conditions are very adverse. What is different about GRAMs is the spatial reasoning capabilities, and the robustness to noise and reverberation. GRAMs natively support binaural localization, and ambisonics localization tasks unlike other models in the literature. Therefore, GRAMs can be used for understanding spatial scenes as well as classifying sounds, or recognizing speech in very noisy and reverberant environments.

Model Details

The GRAM learns spatial audio representation by reconstructing multi-channel masked spectrogram patches. First, a patch extractor consisting of a single convolutional layer with 2D convolutional filters divides each multi-channel spectrogram into n non-overlapping patches. Nonmasked patch embeddings are input to the encoder, for which we selected the 12-layer ViT-Base (ViT-B) Transformer (Dosovitskiy et al., 2021) similar to Huang et al. (2022); Yadav et al. (2024). The encoder outputs patch representations that can be further used for fine-tuning.

Model Description

GRAM is a self-supervised, multi-channel masked auto-encoder model that efficiently learns spatial general-purpose audio representations from simulated real-world sound scenes. To train GRAM, we developed a custom pipeline which makes use of the Soundspace 2.0 platform (Chen et al., 2022a) to simulate high-quality real-world sound scenes from AudioSet (Gemmeke et al., 2017), and of WHAMR! (Maciejewski et al., 2020) for adding background noise. We present two versions of GRAM to ensure flexible application across audio formats: GRAM-Binaural for two-channel audio clips, and GRAM-Ambisonics for four-channel audio clips in the first-order Ambisonics format.

• Developed by: Goksenin Yuksel, [email protected]
• Model type: Transformers, Audio Foundation Models
• Language(s) (NLP): GRAMs support all languages, but mainly English.
• License: MIT

Model Sources

• Repository: https://github.com/labhamlet/GRAM-T
• Paper: https://arxiv.org/abs/2506.00934

Uses

GRAMs can be used as a powerful feature extractor for downstream tasks such as enviromental sound classification, speech recognition, speaker counting, sound localization. Later, training a linear head on top of these extracted features would yield a fine-tuned audio scene analysis model.

How to Get Started with the Model

GRAMs have three strategies to choose from; "raw", "mean" or "cls". We advise to use "raw" strategy as this prooduces embeddings over the time frames, and is more robust to silent parts.

Raw : Break audio clips into non-overlapping 2 second chunks, concatenating the features in time and finally taking a mean over the time axis to generate a fixed vector representation independent of the input audio duration Mean : Break audio clips into non-overlapping 2 second chunks, and taking a mean over all the patches to generate a fixed vector representation independent of the input audio duration Cls : Break audio clips into non-overlapping 2 second chunks, and return the "[CLS]" token representation.

from transformers import AutoModel, AutoFeatureExtractor

model = AutoModel.from_pretrained("labhamlet/gramt-binaural-frame", trust_remote_code=True)
extractor = AutoFeatureExtractor.from_pretrained("labhamlet/gramt-binaural-frame", trust_remote_code=True)

audio = torch.zeros([1,4,320000])
extracted = extractor(audio, return_tensors="pt")
log_mel = extracted['input_values']
print(model(log_mel, strategy = "raw").shape)

Training Details

Training Data

The 85,000 naturalistic scenes were split into a train set of 70,000 scenes (corresponding to 70 Matterport3D houses), and a test set of 15,000 scenes (15 Matterport3D houses) for down-stream evaluation (see Section 3.4). We used the 70,000 naturalistic scenes in the train set to generate naturalistic scenes for all audio clips in the unbalanced training set of AudioSet (10-second sound tracks of 1.74 million YouTube videos (Gemmeke et al., 2017)). Specifically, during training we randomly paired an AudioSet clip with a noise sound clip from the WHAMR! background noise database (Maciejewski et al., 2020). WHAMR! noise clips longer than 10 s were trimmed to 10 s duration and a linear fade-in/fade-out of 200 ms was applied to every noise clip prior to mixing of the sound scene. To create a naturalistic sound scene, we then convolved the AudioSet clip either with BRIR(s, r, θ) for GRAM-Binaural, or with a ARIR(s, r, θ) for GRAM-Ambisonics, to obtain T. Similarly, we convolved the WHAMR! noise clip with the to obtain naturalistic scenes.

Training Procedure

We transformed the channels of each sound scene (i.e., the waveforms) into logscale mel spectrograms using 128 mel filters in the frequency range of 50-16000 Hz with a 25 ms Hanning window and 10 ms hop length, resulting in spectrograms of dimension 1024 × 128. For GRAM-Ambisonic, we extracted normalized active Intensity Vectors (IVs) from the spectrograms as additional input features encoding spatial information. We concatenated mel spectrograms and intensity vectors, resulting in input x = [xmel, IV s] for each naturalistic scene generated from an AudioSet clip. In-batch sampling: As the online mixing of naturalistic acoustic scenes is computationally expensive due to multiple long convolutions, we used a random in-batch sampling procedure to increase the effective batch size in a computationally efficient manner. We randomly sampled 16 partially overlapping segments of 2 seconds to create 16 samples of dimension 200 × 128. This increases the original batch size of 96 to an effective batch size of 1536.

For pre-training, we divided the binaural spectrogram into 2×8×16,ambisonics spectrograms into 7×8×16 patches. We used an adapted version of the mask-based framework of MW-MAE (Yadav et al., 2024), randomly selecting a subset of n patches M1, . . . , Mn for i = 1, . . . , n for masking (masking ratio = 0.8) and replacing their embedding with a learnable mask token. Finally, we added fixed sinusoidal positional embeddings to all embedded patches.

We trained all GRAMs for 500 K steps on an H100 92 GB GPU machine with 16 CPU cores. We used the AdamW optimizer (Loshchilov & Hutter, 2017) with weight decay rate of 0.01, gradient clipping, and a cosine learning rate scheduler with 10 K steps warm-up. The initial learning rate was set to 0.0002, and decayed to 0. We optimize the mean squared error (MSE) loss function between the predicted masked patches and their corresponding input spectrogram patches.

Preprocessing

Firstly, RMS Normalization was applied to audio clips to get all of them in the same loudness levels. Later, instance normalization was applied to the convolved scenes

Training Hyperparameters

• Training regime:: GRAMs were trained with mixed precision, torch.compile and flash attention.

Evaluation

We evaluate GRAM and other state-of-the-art models on the HEAR benchmark task suite, which presents a wide range of tasks to evaluate the downstream performance of audio representation models (Turian et al., 2022). We additionally evaluated performance on simulated real-worldsound scenes using Nat-HEAR.

Testing Data, Factors & Metrics

Testing Data

HEAR: The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. HEAR was launched as a NeurIPS 2021 shared challenge. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.

NatHEAR: Orovides a naturalistic version of all selected datasets in the HEAR benchmark suite in two audio formats: a two-channel, binaural format and a four-channel, first-order Ambisonics format. Weincluded sound localization tasks for two different domains which we generated using HEAR benchmark datasets: A speech localization task based on SC-5, and an environmental sound localization task based on ESC-50. The localization tasks are modeled as a multi-output regression task in which model outputs represent the estimated 3D Cartesian coordinates [x, y, z] on the unit sphere (Adavanne et al., 2018). Finally, to assess the transferability of GRAM to real-world sound scenes, we evaluate also on the sound event detection and localization tasks in TUT Sound Events 2018 REAL (Adavanne et al., 2019)

Results

Summary

We present a General-purpose, Real-world Audio representation Model (GRAM), which learns spatial audio representations using a multi-channel masked auto-encoder approach. GRAM demonstrates remarkable performance in naturalistic sound scenes as well as clean sound scenes, surpassing all state-of-the-art self-supervised spectrogram-based audio foundation models while requiring only a fraction of the training data. Moreover, GRAM is the first audio foundation model that is available in both a two-channel, binaural format and a four-channel, first-order ambisonics format

Model Card Contact

Goksenin Yuksel; [email protected]