voice-openclap-poc

Voice OpenCLAP PoC

This is a proof-of-concept (PoC) release. The model was trained on a single dataset for a few epochs as a rapid feasibility test. It is not intended for production use and will be superseded by a properly trained version.

Model

• Audio encoder: Whisper-small encoder (~88M params, initialized from OpenAI pretrained weights)
• Text encoder: ModernBERT-base (~149M params, pretrained from answerdotai/ModernBERT-base)
• Embedding dimension: 512
• Loss: Symmetric InfoNCE (CLIP-style contrastive loss)

Training

• Dataset: laion/majestrino-data — ~8.2M audio-caption pairs (raw FLAC audio, 32kHz mono, with natural language captions)
• Effective batch size: 1024 (128 per GPU x 8 GPUs)
• Learning rate: 1e-4 cosine schedule, 2000-step warmup
• Weight decay: 0.05
• Precision: amp_bf16
• Logit scale: Clamped at ln(20)
• Samples per epoch: 10M (resampled)
• Best checkpoint: Epoch 3 (this release)

Evaluation Results

Evaluated on a held-out ~21,148 sample validation set from the same dataset distribution.

<details> <summary>Full epoch-by-epoch results</summary>

Overfitting begins after epoch 3 — train loss continues decreasing but val loss and retrieval metrics degrade.

</details>

Usage

import torch
from inference import VoiceOpenCLAP, load_model, preprocess_audio

device = "cuda" if torch.cuda.is_available() else "cpu"
model, tokenizer = load_model("model.pt", device=device)

# Encode text
texts = ["a man speaking calmly", "loud music playing"]
tok = tokenizer(texts, return_tensors="pt", padding=True, truncation=True, max_length=77)
with torch.no_grad():
    text_embs = model.encode_text(tok.input_ids.to(device), tok.attention_mask.to(device))

# Encode audio
wav = preprocess_audio("audio.wav").to(device)
with torch.no_grad():
    audio_emb = model.encode_audio(wav)

# Similarity
sims = audio_emb @ text_embs.T
print(sims)

Requirements

pip install torch torchaudio transformers openai-whisper

Files

• model.pt — Model weights (state dict only, ~0.96 GB)
• inference.py — Self-contained inference script (no external CLAP dependencies needed)
• model_config.json — Model architecture configuration

Disclaimer

This is an early proof-of-concept. Known limitations:

• Trained on a single dataset (Maestrino) with limited diversity
• Begins overfitting after epoch 3 (~30M samples seen)
• Retrieval metrics are modest (4% R@1) — expected to improve significantly with more data, multi-dataset training, and longer schedules
• The mel spectrogram computation requires the openai-whisper package for exact filter bank alignment (a fallback using torchaudio is included but may produce slightly different results)

License

Apache 2.0