paraspeechclap-intrinsic

ParaSpeechCLAP-Intrinsic

ParaSpeechCLAP-Intrinsic is a dual-encoder speech-text model specialized for intrinsic (speaker-level) style attributes that persist across a speaker's utterances, including pitch, texture, clarity, volume, and rhythm. Given a speech clip and a text style description, ParaSpeechCLAP-Intrinsic embeds both into a shared 768-dimensional space and produces a similarity score. It is part of the ParaSpeechCLAP model family from the paper:

ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining
Anuj Diwan, Eunsol Choi, David Harwath
Interspeech 2026

Installation

git clone https://github.com/ajd12342/paraspeechclap.git
cd paraspeechclap
pip install -r requirements.txt

Download the model checkpoint:

mkdir -p checkpoints
huggingface-cli download ajd12342/paraspeechclap-intrinsic paraspeechclap-intrinsic.pth.tar --local-dir checkpoints

Quick Start

Command-line inference

# Compute similarity between audio and a text style description
python scripts/inference.py \
  --checkpoint_path ./checkpoints/paraspeechclap-intrinsic.pth.tar \
  --audio_path /path/to/audio.wav \
  --text "A person speaks in a deep, guttural tone."

# Zero-shot classification across candidate styles
python scripts/inference.py \
  --checkpoint_path ./checkpoints/paraspeechclap-intrinsic.pth.tar \
  --audio_path /path/to/audio.wav \
  --candidates deep shrill nasal husky raspy

Python usage

import torch
import torchaudio
import torchaudio.transforms as T
from paraspeechclap.model import CLAP
from transformers import AutoTokenizer, Wav2Vec2FeatureExtractor

SPEECH_MODEL = "microsoft/wavlm-large"
TEXT_MODEL = "ibm-granite/granite-embedding-278m-multilingual"
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load ParaSpeechCLAP-Intrinsic
model = CLAP(
    speech_name=SPEECH_MODEL,
    text_name=TEXT_MODEL,
    embedding_dim=768,
)
state_dict = torch.load("./checkpoints/paraspeechclap-intrinsic.pth.tar", map_location=DEVICE)
model.load_state_dict(state_dict, strict=False)
model.to(DEVICE).eval()

# Initialize preprocessors
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(SPEECH_MODEL)
tokenizer = AutoTokenizer.from_pretrained(TEXT_MODEL)

# Load and preprocess audio (resample to 16 kHz mono)
waveform, sr = torchaudio.load("path/to/audio.wav")
if sr != 16000:
    waveform = T.Resample(sr, 16000)(waveform)
if waveform.shape[0] > 1:
    waveform = waveform.mean(dim=0, keepdim=True)
audio = feature_extractor(
    waveform.squeeze(0), sampling_rate=16000, return_tensors="pt"
).input_values.to(DEVICE)  # (1, num_samples)

# Similarity with a free-form intrinsic description
text_tokens = tokenizer(
    "A person speaks in a deep, guttural tone.",
    return_tensors="pt", padding=True, truncation=True, max_length=512
)
text_tokens = {k: v.to(DEVICE) for k, v in text_tokens.items()}

with torch.no_grad():
    audio_emb = model.get_audio_embedding(audio, normalize=True)   # (1, 768)
    text_emb = model.get_text_embedding(text_tokens, normalize=True)  # (1, 768)
    similarity = (audio_emb @ text_emb.T).item()
    print(f"Similarity: {similarity:.4f}")

# Zero-shot classification across candidate styles
candidates = ["deep", "shrill", "nasal", "husky", "raspy"]
prompts = [f"A person is speaking in a {s} style." for s in candidates]
tokens = tokenizer(prompts, return_tensors="pt", padding=True, truncation=True, max_length=512)
tokens = {k: v.to(DEVICE) for k, v in tokens.items()}

with torch.no_grad():
    text_embs = model.get_text_embedding(tokens, normalize=True)  # (5, 768)
    scores = (audio_emb @ text_embs.T).squeeze(0)  # (5,)
    pred = candidates[scores.argmax().item()]
    print(f"Predicted style: {pred}")

Best-of-N reranking

Use ParaSpeechCLAP-Intrinsic as an inference-time reward model to select the best speech clip from N TTS candidates:

python scripts/best_of_n.py \
  checkpoint_path=./checkpoints/paraspeechclap-intrinsic.pth.tar \
  input_base_dir=/path/to/tts_outputs \
  output_dir_name=best_of_N_paraspeechclap_intrinsic

Evaluation

Evaluate on the paraspeechclap-eval-intrinsic benchmark:

# Retrieval (R@1, R@10, Median Rank)
python scripts/evaluate_retrieval.py \
  --config-name eval/retrieval \
  checkpoint_path=./checkpoints/paraspeechclap-intrinsic.pth.tar \
  data.dataset_name=ajd12342/paraspeechclap-eval-intrinsic \
  data.audio_root=/path/to/audio_root \
  meta.results=./results_retrieval/paraspeechclap-eval-intrinsic/ajd12342-paraspeechclap-intrinsic

# Per-attribute classification (UAR, Macro F1)
for attr in clarity pitch rhythm texture volume; do
  python scripts/evaluate_classification.py \
    --config-name eval/classification/${attr} \
    checkpoint_path=./checkpoints/paraspeechclap-intrinsic.pth.tar \
    data.audio_root=/path/to/audio_root \
    meta.results=./results_classification/paraspeechclap-eval-intrinsic/ajd12342-paraspeechclap-intrinsic/${attr}
done

Choosing a Model

Related Resources

• GitHub Repository: https://github.com/ajd12342/paraspeechclap
• Models: ajd12342/paraspeechclap-intrinsic, ajd12342/paraspeechclap-situational and ajd12342/paraspeechclap-combined
• Training Dataset: ajd12342/paraspeechcaps-intrinsic-train
• Parent Dataset: ajd12342/paraspeechcaps
• Evaluation Datasets: ajd12342/paraspeechclap-eval-intrinsic, ajd12342/paraspeechclap-eval-situational, ajd12342/paraspeechclap-eval-combined

Citation

@misc{diwan2026paraspeechclapdualencoderspeechtextmodel,
      title={ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining},
      author={Anuj Diwan and Eunsol Choi and David Harwath},
      year={2026},
      eprint={2603.28737},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2603.28737},
}