vocal-burst-detector-x2

vocal-burst-detector-x2

Which file do I want?

production/ — start here. The instrument this repo exists to ship: a 5-member ensemble on the laion/voiceclap-large-v2 encoder, trained on both halves of laion/vocal-bursts-segments, with bs_score.py (an importable scorer), per_class_recall.json (read this before you read any hit rate) and the full confusion matrices. Jump to The production head.

The repo root is a drop-in for vocal_burst_mlp_v2.pt — same frozen 768-d FastScorer features, so it swaps into an existing reward.py without touching the feature path. It is beaten by production/ in every cell and is kept for compatibility, not for accuracy.

voiceclap/ is the five-split-seed research arm behind the encoder choice; its numbers, not its weights, are what you want.

Which of the two heads is better, in plain terms

Both solve the easy question and only half of the hard one. "Is a sound there at all" is essentially done — 97 to 99 % correct. "Which sound is it" is not: over 17 classes the small head gets 38 % and the big one 45 %, against a 5.9 % chance rate. That is six to eight times better than guessing and still a long way from reliable. The mistakes are almost never deafness; they are confusions inside a family — a giggle called a chuckle, an exhausted groan called a frustrated one. Scored at family level both gain 16 to 21 points.

They fail on different classes, and that is the useful part. Per class, trained on both sources and tested on real speech:

large-v2 wins 12 of 17 classes on real speech and 11 of 17 on DramaBox, and the pattern is legible: it wins on the quiet, drawn-out, breathy sounds — inhales, hums, yawns, breathing — which carry little energy and shade gradually into ordinary breath. The 768-d head wins on the loud, short ones, and most sharply on Scream, 0.768 against 0.576.

That exception matters more than its size suggests. Scream is the class this project has spent the most effort on, and a detector decides whether a hit counts as a hit at all — so grading burst adapters with the big head systematically understates screams. Note also that both heads are simply unusable on some classes: Deep Breath 0.032 on the small head, Relief Sigh 0.088 and 0.128 on the two.

Recommendation. Use the 768-d head as the default — it rides along for free and is better at the sound that matters most here. Reach for production/ when the target is breathing, humming or yawning, or when the question is only "is anything there". The most honest use is to run both and mark the disagreements as uncertain: that they fail in opposite directions on five of seventeen classes is the single most useful thing this comparison produced.

A 17-class vocal-burst head — including a no_burst reject class, which laion/vocal-burst-detector-v2 has but was never trained against verified negatives — on top of that detector's own frozen 768-d feature extractor. It is a drop-in replacement for the shipped vocal_burst_mlp_v2.pt: same embedder, same 768 → 256 → C architecture, new output layer.

Five seeds ship. vocal_burst_mlp_x2_s{0..4}.pt each hold arch, classes and state_dict.

What it is measured to do, and why the measurement is the cross-source one

A within-source split cannot answer "how robust is it": training and test labels both come from gemini-3.8-flash, so the model is graded by the standard it was trained on. The informative measurement trains on one generator and tests on the other. Real speech and DramaBox TTS have nothing acoustic in common.

Chance = 5.9 % over 17 classes. Five seeds, grouped splits (real: by speaker; DramaBox: by prompt, so all three seeds of one sentence move together). The test set for a source is fixed per seed and reused by every arm.

neg speech and neg silence are reported separately and never pooled. Silence is a trivially easy negative; in production every decision is speech-vs-burst. Where the two disagree, the speech column is the real one.

Per class

Use

import torch, torch.nn as nn
ck = torch.load("vocal_burst_mlp_x2_s0.pt", map_location="cpu", weights_only=False)
a = ck["arch"]
net = nn.Sequential(nn.Linear(a["D"], a["H"]), nn.BatchNorm1d(a["H"]), nn.GELU(),
                    nn.Dropout(a["dropout"]), nn.Linear(a["H"], a["C"]))
net.load_state_dict({k.replace("net.", ""): v for k, v in ck["state_dict"].items()})
# features: FastScorer.emb.encode_waveform on a 16 kHz mono cut, exactly as vocal-burst-detector-v2

The production head — production/

This is the instrument to re-score with. Encoder laion/voiceclap-large-v2 (3584-d, frozen), head 3584 → 256 → 17, five initialisations averaged in probability space, trained on both halves of laion/vocal-bursts-segments. The root of this repo is still the FastScorer drop-in; production/ is the better instrument and the one that should be used for new measurements.

An ensemble across split seeds would be dishonest to score — seed k's member has trained on seed j's test items — so the ensemble uses one grouped split, five initialisations inside it, and is scored on that split's held-out groups, which no member has seen.

Chance is 5.9 % over 17 classes. The all held-out rows follow the corpus's own class prior and are the right rows for reading recall; the balanced rows are the ones comparable to the cross-source table above.

⚠ Read the per-class recall before reading any hit rate

A hit-rate table that says "burst class X is produced at rate H" is measuring two things at once: the generator producing the sound, and this detector being able to name it. They multiply. A class this detector recalls at 20 % cannot show a hit rate meaningfully above 20 %, however good the generator is — so without the recall column, a low row is unreadable: you cannot tell a silent generator from a deaf instrument. With it you can: a hit rate near its class's recall means the detector was the limit; far below it means the generator was.

production/per_class_recall.json and .csv ship for exactly this. Classes marked ⚠ have fewer than 30 held-out examples in one of the two sources and their recall is not a solid estimate.

Family recall is beside strict recall on purpose. The failure mode here is granularity, not deafness — Breathy Giggle → Chuckle, Exhausted Groan → Frustrated Groan. A recipe that reliably produces a groan but gets named the wrong groan is useful, and reporting it as a failure without the family column would be wrong.

How to call it

import sys; sys.path.insert(0, "<dir containing bs_score.py>")   # ships in production/
from bs_score import ProductionBurstScorer

sc  = ProductionBurstScorer(device="cuda", ckpt_dir="<.../production>")
out = sc.bursts(wavs16)          # list[np.float32 @16 kHz] -> list[list[dict]]

out[i] is the bursts in clip i, in the same shape reward.RewardModel.bursts returns, so gl_run.py / vc_run.py-style code swaps instrument without changing its aggregation:

{"start": 3.12, "end": 3.74, "dur": 0.62,
  "label": "Scream", "prob": 0.83,
  "labels": [["Scream", 0.83], ["Shriek", 0.07], ["Frustrated Groan", 0.03]],
  "family": "scream", "no_burst_prob": 0.02}

• sc.bursts(wavs16, no_burst_gate=0.5) — locate with laion/vocalburst-locator v2 (unchanged constants, so a span is the same span the old stack reports; only the name changes), then name, dropping spans the head rejects.
• sc.classify(cuts16) — name spans you already cut; drops nothing, apply your own gate.
• sc.recall_floor() — the table above as {class: {source: …}}.

python bs_score.py runs a self-test.

A second encoder was tried: laion/voiceclap-large-v2

Single-variable arm — same segments, same grouped splits, same head shape, same five seeds; only the encoder changes. The dimension changes with it, so this head is 768 → 256 → 17 while the VoiceCLAP one is 3584 → 256 → 17, and the VoiceCLAP arm has no shipped-detector baseline and no warm start because that detector cannot consume a 3584-d vector.

Sanity check on the same 120 segments before either was trained on: 1-NN class accuracy 78.3 % (VoiceCLAP) against 74.2 % (FastScorer), chance 25 %. VoiceCLAP throughput was measured at 15.6 segments/s on one GPU (3.1 GPU-hours for the whole corpus).

One conclusion changes with the encoder and it is the most useful thing here: with the 768-d extractor, adding the DramaBox half cost 5.2 points on real speech (both→real 38.2 % against real→real 43.4 %); with VoiceCLAP the same comparison is 44.9 % against 45.6 % — under a point. The harm from mixing the two sources was largely an encoder limitation, not a property of the data. The one place VoiceCLAP is worse: a head trained on DramaBox alone recognises real-speech negatives less well (84.1 % against 95.8 %) — the richer encoder also encodes more of the generator.

Both ship. The root of this repo is the FastScorer head — the drop-in for vocal_burst_mlp_v2.pt, consuming exactly the vectors reward.py already computes. voiceclap/vocal_burst_mlp_vclap_s{0..4}.pt are the VoiceCLAP heads (3584 → 256 → 17); they need laion/voiceclap-large-v2 at inference (~18 GB, ~16 segments/s on one GPU), which is the price of the extra points.

A third encoder, and the finding that it is not a third encoder: laion/voiceclap-commercial

commercial/ holds the same head trained on embeddings from laion/voiceclap-commercial, the 110 M dual-tower encoder the demo server already loads for retrieval. It was trained to give the burst term of the best-of-N reward an encoder that costs nothing extra to run.

It turned out to be the encoder this repository was already using. The head above is described as running on "the shipped detector's own 768-d extractor" (FastScorer), and that name hides which model it is. Measured directly on the stored feature vectors — 300 segments present in both feature sets, same keys, same order:

So they are the same encoder, differing only by float non-determinism and batch composition. The accuracy differences below (0.4334 against 0.4344 on real→real) are re-training noise between two runs, not an encoder effect, and should not be read as one being better.

What this is good for anyway: it makes the encoder explicit and citable. A consumer that wants to compute the features itself can now load a named public model instead of an internal class whose provenance is not stated, and the demo server's BON_BURST_ENCODER = "commercial" is satisfied by a head in this repo rather than by an assumption.

The four arms side by side

Balanced accuracy over 17 classes, 5 seeds, chance = 5.88 %. Each row is a cross-source cell: trained on the left, tested on the right.

Burst versus no-burst, same cells:

How to choose between them, which is a cost question rather than an accuracy question once the identity above is understood:

• commercial / the default head — 768-d, already resident in any process that does retrieval, so the marginal cost of a burst score is one 768 → 256 → 18 MLP on an embedding that exists anyway. This is the drop-in.
• voiceclap/ (large-v2) — 3584-d, wins every one of the six cells, by +1.4 to +6.8 points on class accuracy and by 1 to 3.5 points on burst-versus-no-burst. It is a 7 B model and a second encoder on the card: 3.1 GPU-hours to embed 175 k segments at 15.65 segments/s, peak 24.9 GiB.
• small-v2 — 768-d and genuinely different, unlike commercial. It is better than either 768-d option on DramaBox audio (0.3934 against 0.3416 on real→dramabox, 0.5553 against 0.5045 on dramabox→dramabox) and no better on real speech. If the target is synthetic audio it is the better small encoder; on real speech it is not.

One exception worth stating, because it cuts against the big model. On dramabox→real, large-v2's accuracy on the speech negatives falls to 0.8407 while every 768-d arm holds 0.94–0.96. A richer encoder also encodes more of the generator, so a head trained on a single source overfits that source harder. If you train on one source only, the small encoder is the safer one.

What it cannot do

• It is a classifier, not a locator. It labels a span something else has proposed. The spans it was trained on are gemini-3.8-flash's, which this project measured as better at what and worse at where than the shipped locator.
• Its labels have never been checked by a human. Every training and test label is one model's opinion. A shared prior between that model and the detector whose features this uses would inflate every number above and nothing measured here can see it.
• Half its training data is TTS output. The DramaBox half of laion/vocal-bursts-segments is generated audio whose LTX-2 Community Licence has not been assessed; if that matters for your use, retrain from the real config alone — the real->real and real->dramabox rows above are what that head scores.

<!-- vocal-burst-groups-23:start -->

Class groups

vocal_burst_groups.json and GROUPS.md carry a 23-group scheme over 117 burst label strings, grouping names that denote the same or a very similar sound (snicker/chuckle, shriek/scream, cough/coughing). Scoring the same predictions at group level raises the mean generation hit rate from 0.302 to 0.537; a random grouping with identical group sizes reaches 0.355, so +0.182 of it is the grouping being right and the rest is arithmetic. Groups were checked with directed lift rather than raw confusion, because two labels account for 29 % of all annotator top-1 calls whatever was requested and merging on raw confusion books a generation failure as a hit.

For training the classifier, keep the fine classes: collapsing them raises raw accuracy only because chance rises with it. Group at evaluation time — that can be done at any point, the reverse cannot.

<!-- vocal-burst-groups-23:end -->