vocalburst-classifier-multilabel

VocalBurst Classifier — multi-label

A presence classifier for non-speech vocal bursts (laughs, sighs, gasps, coughs, groans, throat sounds, …) over the 82-class LAION VocalBurst taxonomy + no_burst. Multi-label (independent sigmoid per class) — a clip can contain several bursts. Works on pure bursts and bursts embedded in speech.

Architecture

laion/voiceclap-commercial (768-d, frozen) → MLP 2048-wide × 4 deep (LayerNorm+GELU+dropout) → 83 sigmoids. ~14.4M trainable params.

Training

• Positives: Gemini-confirmed single bursts (DACVAE, 82 classes) + real speech-with-bursts (gemini_pro_25, multi-label).
• Negatives / speech carriers: Emilia clips filtered burst-free by laion/vocalburst-locator.
• ~97k samples: pure bursts + composites (burst + speech carriers + a 2nd burst) + 25% overlays (burst mixed over speech) + neg+neg. The speech carriers carry no label, so the head must localise the burst, not memorise audio. Balanced (≤1000/class); 50 epochs, AdamW, BCE with per-class pos-weight, best-mAP checkpoint.

v2 (current): retrained with 9,000 multilingual burst-free negatives (FLEURS: Chinese, Hindi, Bengali, Arabic, Persian, Urdu, Tamil, Telugu, Vietnamese, Thai, Indonesian, Japanese, Korean, Swahili, Yoruba, Zulu, Turkish, Russian) — fixes hallucinated bursts on non-European speech.

Results (Gemini-mixed validation)

How it works

Two stages: a frozen laion/voiceclap-commercial audio encoder turns a clip into a 768-d embedding (encode_waveform, auto-downloaded — the repo needs no extra setup), then this small trained MLP head maps it to 83 outputs = 82 VocalBurst classes + no_burst (taxonomy: LAION-AI/voice-taxonomies · vocalburst). A no-burst gate: if P(no_burst) ≥ 0.5 the clip is declared burst-free (no false alarm); otherwise the top classes are returned. Clips are truncated to the first 30 s (the encoder's window).

Usage

from inference import VocalBurstClassifier
clf = VocalBurstClassifier("laion/vocalburst-classifier-multilabel")      # HF repo id, or a local checkout dir
print(clf.predict("clip.wav"))        # -> {no_burst, p_no_burst, top1, predictions:[(class,prob)], group}

Files

model.pt (MLP weights) · config.json (arch) · classes.json (83 labels, index order) · class_to_group.json (fine→16 coarse families) · inference.py · example.py · requirements.txt.

Interactive demos (audio + predictions)

• Multi-label predictions vs ground truth: https://projects.laion.ai/procedural-voice-captions/vocalburst-predictions/
• Single-burst classifier: https://projects.laion.ai/procedural-voice-captions/vocalburst-single/
• Two-stage detect→classify combo: https://projects.laion.ai/procedural-voice-captions/vocalburst-combo/
• On MOSS character voices: https://projects.laion.ai/procedural-voice-captions/vocalburst-character/

Training data + embeddings: laion/vocalburst-classification. Embedder: laion/voiceclap-commercial. License: CC-BY-4.0.