voiceclap-vocalburst-blend

VoiceCLAP Vocal-Burst Blend Scorer

A tiny, self-contained model that predicts, from a short speech clip containing a vocal burst (laugh, sob, gasp, sigh, groan, scoff, etc.), a 0-10 score for how naturally that vocal burst blends into the surrounding speech.

It is designed as a fast, cheap proxy reward for evaluating expressive TTS / voice-acting systems: does the non-verbal burst sound like an organic part of the performance, or does it sound spliced-in, robotic, or emotionally mismatched?

The package is fully standalone. It bundles the frozen VoiceCLAP-small audio embedder (weights + custom modeling code) and the trained blend heads, so no other repositories are fetched at inference time.

What it predicts — the 0-10 rubric

Intermediate values interpolate between these anchors. The output is clamped to [0, 10].

Which model to use

This repo ships two recommended heads plus a legacy one. All are tiny MLPs on top of the same frozen 768-d VoiceCLAP-small embedding.

base_d0.2 is the default because it has the best MAE and correlation on a decontaminated, class-balanced test set (see the evaluation story below).

Architecture

16 kHz mono waveform
      │
      ▼
VoiceCLAP-small audio encoder   (frozen, 768-d embedding)
      │  encode_waveform → 768-d
      ▼
standardize:  z = (emb − μ) / σ        (μ, σ are 1×768 stats from training)
      │
      ▼
Blend head (MLP):
   Linear(768 → H) → GELU → Dropout → Linear(H → 1)    (H = 25 base, 50 big)
      │
      ▼
blend score ∈ [0, 10]

• Embedder: VoiceCLAP-small, a dual-tower CLAP-style model. Only the audio tower (encode_waveform) is used and it is kept frozen. It is bundled in this repo under voiceclap_small/ and loaded locally via trust_remote_code=True — nothing is downloaded at inference time.
• Head: a small MLP (≈19k params for base, ≈38k for big) trained on top of the frozen 768-d embeddings. μ/σ standardization stats are stored inside each checkpoint.

Training data

The two recommended heads (base_d0.2, big_d0.2) were trained on a scaled dataset of 15,843 clips:

• 14,389 clips containing a vocal burst, each labeled 0-10 for blend naturalness by Gemini-3.1-Pro.
• 1,454 clips confirmed by a burst detector to contain no burst, labeled 0 as negatives.

The training objective is a Huber regression loss on the 0-10 target with a frozen encoder (only the MLP head is trained).

The legacy v1 head was trained on the earlier ~1,600-clip dataset.

The honest evaluation story

The small validation set was misleading. On the ~110-clip held-out validation split, the original v1 model looked better (val MAE ≈ 2.07) than the new scaled models (val MAE ≈ 2.38). But that val split is small and drawn from the same distribution the model overfits to.

To get an honest read we built a decontaminated test set of 275 clips (25 per blend value, 0-10), with no overlap with training/validation. On this balanced, decontaminated test the picture reverses:

Takeaway: scaling the data materially improved generalisation (lower MAE, higher correlation on the decontaminated test) even though it looked worse on the tiny in-distribution val set — a textbook case of the small val set rewarding overfitting.

See blend_models_report.html for the full interactive results: the complete architecture/dropout/mixture ablation, the decontaminated-test breakdown, and a sortable table comparing the two models on the validation set (with embedded audio you can play).

Usage

from blend_model import BlendScorer

# Loads the bundled VoiceCLAP-small embedder + blend head, all local.
# model defaults to "base_d0.2" (best generaliser); also "big_d0.2" or "v1".
scorer = BlendScorer(pkg_dir=".", model="base_d0.2", device="cuda")  # or device="cpu"

# Score a single wav (any sample rate / channel count; resampled to 16k mono):
score = scorer.score("clip_with_laugh.wav")
print(f"blend naturalness: {score:.2f} / 10")

# Score many at once:
scores = scorer.score_batch(["a.wav", "b.wav", "c.wav"])

# Score an in-memory waveform:
import torchaudio
wav, sr = torchaudio.load("clip.wav")
score = scorer.score_waveform(wav, sr)

Command-line:

pip install -r requirements.txt
python example.py clip_with_laugh.wav base_d0.2
# -> [base_d0.2] blend score (0-10): 7.42

Files in this repo

Limitations

• Frozen encoder caps accuracy. Quality is upper-bounded by what the frozen VoiceCLAP-small audio embedding captures; the head cannot recover information the embedding discards.
• Mid-range scores are hardest. The model is most reliable at the extremes (clearly organic vs. clearly spliced/absent). Scores around the middle of the scale (4-6) carry the most uncertainty.
• Use as a fast proxy, not a judge. This is a lightweight reward / filtering signal (e.g. for ranking or reward-shaping TTS outputs). It is not a replacement for a strong multimodal judge or human evaluation on high-stakes decisions.
• Domain. Trained on speech clips with vocal bursts; behaviour on music, noise-only audio, or non-speech is undefined.

License

CC-BY-4.0. You are free to share and adapt these models, including for commercial use, provided you give appropriate credit to LAION. See https://creativecommons.org/licenses/by/4.0/.

Created and released by LAION.