audioforge-ast-fsd50k

audioforge-ast-fsd50k

LoRA-adapted Audio Spectrogram Transformer (AST) trained on FSD50K for multi-label environmental sound event classification, produced by the AudioForge training pipeline.

Model Details

• Base model: MIT/ast-finetuned-audioset-10-10-0.4593 (Audio Spectrogram Transformer, pretrained on AudioSet)
• Adaptation method: Low-Rank Adaptation (LoRA) via peft library
  • Rank (r): 8
  • Alpha: 16
  • Dropout: 0.05
  • Target modules: ["q_proj", "v_proj"]
• Trainable parameters: 450,248 out of 86,792,848 total (0.52%)
  • Backbone frozen, only LoRA adapters + 200-class classifier head trained
• Task: Multi-label audio event classification, 200 sound classes (FSD50K)
• Loss: Sigmoid + Binary Cross-Entropy (multi-label)
• Training: 5 epochs, single NVIDIA A10G GPU (AWS g5.xlarge), ~71 minutes, fp16 mixed precision
• Validation split: 4,170 samples

Validation Metrics (FSD50K)

Usage

This repository contains only the LoRA adapter (adapter_config.json + adapter_model.safetensors, ~1.8 MB). The base model must be downloaded separately and combined with this adapter at load time.

from transformers import AutoModelForAudioClassification, AutoConfig
from peft import PeftModel

# Load base model configuration and set for FSD50K multi-label task
base_config = AutoConfig.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593")
base_config.num_labels = 200
base_config.problem_type = "multi_label_classification"

# Load base model (87M parameters, ~340 MB)
base_model = AutoModelForAudioClassification.from_pretrained(
    "MIT/ast-finetuned-audioset-10-10-0.4593", 
    config=base_config, 
    ignore_mismatched_sizes=True
)

# Apply LoRA adapter
model = PeftModel.from_pretrained(base_model, "auro-rirum/audioforge-ast-fsd50k")
model.eval()

# Input: mel-spectrogram [batch, max_length, num_mel_bins] from transformers.AutoFeatureExtractor
# Usage: logits = model(input_values=input_values).logits
#        probs = torch.sigmoid(logits)  # Multi-label probabilities for each class

Dataset

FSD50K (Freesound Dataset 50K): Large-scale, high-quality annotated dataset of environmental audio, with 50,197 audio clips labeled across 200 sound event classes. Multi-label annotations allow clips to have multiple simultaneous events.

Notes

• This is an adapter-only repository. To use this model, you must download the base model from Hugging Face.
• The adapter was trained with mixed-precision (fp16) for efficiency; inference works in both fp32 and fp16.
• Performance varies across classes; strong performance on broad categories (Music, Animals, Explosions) and weaker on rare or semantically ambiguous classes.