vital-jepa-v4

Vital Audio JEPA V4

V4 trains the JEPA representation around continuous acoustic properties rather than treating PresetShare labels as the main objective. It uses 74,627 stereo renders produced by Vital 1.6.4 from 7,394 unchanged parent presets.

Input

• 22,050 Hz stereo PCM16 audio, exactly 6.0 seconds
• four-second held note plus two-second release tail
• no waveform peak or loudness normalization
• energy-preserving mid/side conversion
• two-channel, 128-bin log-mel spectrogram
• 1024-sample FFT, 256-sample hop, and 16x16 patches

Feature-focused objectives

The pooled representation predicts 15 normalized continuous descriptors:

• level, peak, crest factor, attack, tail energy, and tail/body ratio
• spectral centroid, bandwidth, brightness, flatness, and flux
• modulation depth
• stereo width, correlation, and channel imbalance

Training also includes:

• frequency-band, long-time-span, random-block, and release-tail masks
• side-channel context masking with full stereo targets
• balanced parent/category sampling
• different-note positive pairs from unchanged presets
• center-note parent/mutation pairs
• regression and cosine-direction losses for measured acoustic changes caused by parameter mutations

Taxonomy supervision remains a low-weight diagnostic. The old generic recipe classification and raw parameter-delta losses have zero weight in V4.

Training

• 36,302,000 parameters total; 22,105,520 trainable
• 70,916 clips / 7,022 parents for training
• 1,859 clips / 186 parents for validation
• 1,852 clips / 184 parents for the untouched test split
• batch size 64, bfloat16, AdamW
• 20 epochs / 22,160 optimizer steps
• RTX PRO 4500 Blackwell
• full training time: 1,980.5 seconds (33.0 minutes)

The final step was selected by maximum held-out mean acoustic-feature correlation, with cross-parent retrieval as the secondary criterion.

Untouched parent-held-out results

Every retrieval query excludes all variants sharing its parent preset.

Selected descriptor correlations include 0.972 for RMS level, 0.943 for spectral centroid, 0.905 for modulation depth, 0.906 for stereo width, and 0.905 for stereo correlation.

Limitations

Spectral flux (0.579 correlation), channel imbalance (0.491), and mutation direction prediction (0.283 cosine) remain substantially weaker than the core level, spectrum, envelope, and stereo features. The descriptors are practical signal-analysis targets, not a complete model of human timbre perception.

This encoder is intended for retrieval and as an initialization for editable Vital sound design. It does not directly recover a complete synthesizer preset.

checkpoints/jepa_v4/selected.pt is the optimizer-free selected checkpoint. The evaluation reports and training log are included for reproducibility.