voicenet-dimension-predictors-commercial

VoiceNet Dimension Predictors (VoiceCLAP-commercial)

57 per-dimension VoiceNet predictors on top of frozen VoiceCLAP-commercial (768-d, L2-normalized) audio embeddings. For every one of the 57 VoiceNet perceptual voice dimensions this repo ships two lightweight MLP heads:

• a regression head (continuous score on the dimension's 0-k scale), and
• a classification head (discrete level/bucket + confidence).

One commercial encode -> all 57 dimensions. Embed a clip once with the bundled VoiceCLAP-commercial encoder, then run all 114 heads (57x2) on that single 768-d vector.

This repo is self-contained: it bundles the full VoiceCLAP-commercial embedder under voiceclap_commercial/, so no separate model download is needed.

How to read these numbers (plain language)

Each dimension is scored on an ordinal 0-k scale (k=6 for most, so 0..6; a few have fewer levels). We report, on a held-out validation set the model never trained on:

• MAE (regression) - Mean Absolute Error: on average, how many scale-points the predicted score is away from Gemini's score. Lower is better. MAE 0.5 means predictions land about half a level off on a 0-6 scale; MAE 1.3 means more than a full level off.
• Pearson r (regression) - how well predictions track the true ranking (correlation), from -1 to 1. Higher is better. ~0.9 = predictions move almost in lockstep with the labels; ~0.5 = only a weak trend.
• within-1 (regression) - the fraction of clips whose predicted score is within +-1 level of the true score. Higher is better (0.90 = 90% of predictions are at most one level off).
• accuracy (classification) - fraction of clips whose predicted discrete bucket exactly matches Gemini's. Higher is better. Chance is ~1/k, so ~0.14 for a 7-level axis; 0.60 is far above chance.

So a strong dimension looks like low MAE + high Pearson + high within-1 + high accuracy.

What are the dimensions?

The 57 VoiceNet dimensions cover arousal, valence, gender, age, tempo, brightness, resonance (chest/head/nasal/...), speaking styles (news, ASMR, dramatic, ...), recording quality, and more. See dimensions.py for every code, human name, level count, and per-level definition.

How it was trained (round-1+2+3, best-variant-per-dimension)

• Labels: laion/emolia-voicenet-gemini-annotations - Emolia clips each re-scored on their single VoiceNet dimension by Gemini-3.5-flash. This retrain uses the combined round-1 + round-2 + round-3 labels (round 3 topped up the most data-starved score levels of the hardest dimensions).
• Frozen validation split: the same held-out validation ids as the previously published run (val_ids.json), so every number below is directly comparable to the round-1+2 release.
• Embeddings: each unique clip encoded once with VoiceCLAP-commercial (768-d, L2-normalized).
• Three balancing variants per dimension - cap3x / cap4x / cap5x. Samples are bucketed by Gemini score; each bucket is capped at 3x / 4x / 5x of S (S = the count of the second-smallest usable bucket), larger buckets randomly subsampled to that cap. Near-empty buckets (<4) are skipped. The validation split is frozen and identical across variants.
• Heads / losses: regression = HuberLoss (delta=1.5), H=64, dropout=0.33; classification = CrossEntropy, H=48, dropout=0.33. Adam, 50 epochs, input standardized by train mean/std, best-val checkpoint kept (val MAE for reg, val accuracy for cls).
• Best-variant-per-dimension selection: for each dimension we keep the variant that did best on the frozen val set - the regression head with the lowest val MAE, and the classification head with the highest val accuracy (chosen independently). Those are the top-level regression/ + classification/ defaults that predict.py loads. All 6x57 checkpoints are also shipped under variants/{cap3x,cap4x,cap5x}/ for reproducibility.

Variant comparison (mean over all 57 dimensions)

Best-variant picks (how many dimensions selected each variant):

• Regression (min val MAE): cap3x: 18, cap4x: 15, cap5x: 24
• Classification (max val acc): cap3x: 28, cap4x: 15, cap5x: 14

The best-per-dimension defaults (mixing variants) reach mean reg MAE 0.744, mean Pearson 0.79, mean within-1 0.870, mean cls accuracy 0.617, mean cls within-1 0.863.

Did round-3 data help? (before / after)

Against the previous round-1+2 published defaults (regression = cap4x, classification = cap2x), on the identical frozen validation split:

Top-5 dimensions most improved on MAE (per-dim delta vs the round-1+2 cap4x regression MAE): R_MIXD (Mixed Resonance, 0.920→0.787, Δ0.133), R_NASL (Nasal Resonance, 0.780→0.683, Δ0.096), COGL (Cognitive Load, 1.131→1.052, Δ0.079), R_THRT (Throat Resonance, 0.797→0.718, Δ0.078), S_CART (Cartoonish Style, 1.191→1.115, Δ0.076).

Per-dimension metrics (default = best variant, sorted by MAE)

• Strongest (lowest MAE): EXPL, R_HEAD, BKGN, SMTH, VOLT.
• Weakest (highest MAE): FOCS, VFLX, S_RANT, ARSH, S_ASMR.

The weakest axes are the fine resonance/timbre and shift/flux dimensions (e.g. R_MIXD, R_NASL, arousal/valence-shift, velocity-flux): the Gemini labels for these are heavily collapsed into one or two buckets (VoiceCLAP zero-shot bucketing cannot separate them), so there is little balanced signal to learn and their errors are inherently larger. Coarse perceptual axes (arousal, valence, gender, formality, tempo, brightness, background noise) predict well.

Inference

from predict import VoiceNetPredictor
p = VoiceNetPredictor()             # bundled embedder + best-per-dim heads (114 total)
out = p.predict("audio.wav")        # one encode -> 57 dims
print(out["AROU"])   # {'name': 'Arousal', 'reg_score': .., 'cls_bucket': .., 'confidence': .., ...}

# force one balancing variant instead of the best-per-dim defaults:
p4 = VoiceNetPredictor(variant="cap4x")

or from the shell: python predict.py audio.wav (add --variant cap4x to force a variant).

Each dimension returns the regression score, the classification bucket, and the softmax confidence.

Files

• regression/<DIM>.pt, classification/<DIM>.pt - 57 + 57 best-per-dimension default heads. Each: {state_dict, mu, sd, arch, dim, name, levels, val_metrics}.
• variants/{cap3x,cap4x,cap5x}/{regression,classification}/<DIM>.pt - all 6x57 variant heads.
• voiceclap_commercial/ - the full bundled VoiceCLAP-commercial embedder (self-contained).
• dimensions.py, predict.py, requirements.txt.
• metrics.parquet - per dimension x variant validation metrics.
• metrics_best_per_dim.parquet - the chosen best variant per dimension per head + its metrics.

Notes & limitations

• Labels are Gemini-3.5-flash perceptual estimates from a single flash-LLM annotator, not ground truth.
• Fine resonance/timbre axes are collapsed in the labels and predict near-constant - expected.
• The bundled embedder is included for self-containment; heads run on its 768-d L2-normalized output.

Project & Taxonomy

These predictors are the models of the VoiceNet project — predicting how speech is delivered across 57 perceptual voice/prosody dimensions (0–6).

• GitHub project & docs: https://github.com/LAION-AI/voicenet — self‑contained documentation, including a full inference guide (throughput optimization, WebDataset/mp3 loading, multi‑GPU scaling) and training guide.
• Live demo: https://projects.laion.ai/voicenet/ — these predictors on 100 diverse multilingual Emolia clips.
• Taxonomy: every dimension's code, name, level count, and per‑level 0–6 definition ships in this repo as dimensions.py; the full narrative rubric is on GitHub as taxonomy/voicenet_taxonomy.md.

The VoiceNet taxonomy and project come from the forthcoming VoiceNet paper (not yet published).

Links

• Label source: laion/emolia-voicenet-gemini-annotations
• Base embedder: laion/voiceclap-commercial
• VoiceNet project: https://github.com/LAION-AI/voicenet

License

CC-BY-4.0