vital-jepa-v3

Vital Audio JEPA V3

V3 adds semantic and preset-aware training signals to the masked audio JEPA encoder. It is trained on 74,685 parameter-aware renders produced by Vital 1.6.4 from 7,394 unchanged parent presets.

Input

• 22,050 Hz mono PCM audio
• exactly 6.0 seconds
• 128-bin log-mel spectrogram
• 1024-sample FFT and 256-sample hop
• 16x16 spectrogram patches: an 8x32 grid with 256 tokens

V3 changes

• category- and parent-balanced batches
• guaranteed pairs of different MIDI-note renders from the same parent
• cross-parent supervised contrastive learning over a cleaned seven-class taxonomy
• note-invariance contrastive learning only for performance variants
• auxiliary coarse-category, mutation-group, and mutation-delta heads
• parent-disjoint train, validation, and test splits

The coarse taxonomy merges related noisy PresetShare labels into bass, drums, fx, keys, lead, pad, and rhythm.

Training

• 35,894,930 parameters total; 21,698,450 trainable
• 70,974 clips / 7,022 parents for training
• 1,860 clips / 186 parents for validation
• 1,851 clips / 184 parents for the untouched V3 test split
• batch size 64, bfloat16, AdamW
• 20 epochs / 22,160 optimizer steps
• RTX PRO 4500 Blackwell
• full training time: 1,928.8 seconds (32.1 minutes)

Step 4,000 was selected using validation retrieval. Later checkpoints reduced masked JEPA loss but did not improve semantic retrieval.

Untouched parent-held-out test results

All retrieval results exclude every variant sharing the query's parent.

The V3 coarse-category head reaches 62.45% accuracy on the same unseen test parents. The experimental recipe-group head does not generalize: it predicts the performance class for the test set, yielding 29.34% accuracy. Do not use that head as a reliable mutation classifier. The representation and category results remain valid, and the failed auxiliary result is included explicitly in the evaluation report.

Intended use

This encoder is a retrieval and initialization baseline for editable Vital sound design. It is not an exact audio-to-preset reconstruction model. Coarse category retrieval is only a proxy for perceptual usefulness, and sparse fine labels such as vox, drone, and reese remain weak.

checkpoints/jepa_v3/selected.pt is the optimizer-free selected checkpoint. See eval_runs/jepa_v3/ for checkpoint selection, coarse/fine test reports, controlled V2 baselines, embeddings, and the full training log.