wav2vec2-large-xlsr-deepfake-audio-classification

Wav2Vec2 Large XLSR - Deepfake Audio Classification (ONNX)

This is the ONNX format of Gustking/wav2vec2-large-xlsr-deepfake-audio-classification, a Wav2Vec2 model fine-tuned for binary classification of audio as real (bonafide) or fake (spoof/deepfake).

The model was converted from PyTorch (safetensors) to ONNX for use with ONNX Runtime, enabling deployment without a PyTorch dependency. Three variants are provided to suit different size/accuracy tradeoffs.

Available Variants

All three variants produce identical classification decisions across all test samples. Choose based on your deployment constraints:

• FP32 — baseline, bit-for-bit closest to the original PyTorch model.
• FP16 — best balance of size and accuracy. Weights stored as float16, cast to float32 at runtime. Virtually no accuracy loss (0.05% relative error).
• INT8 — smallest model. Only MatMul (attention/FFN) weights are dynamically quantized to INT8; the feature extractor Conv layers are kept in full precision to preserve accuracy.

Model Details

Original Model Performance

Results on ASVspoof2019 evaluation subset:

Accuracy Validation (vs PyTorch, 20 samples)

All variants were validated against the original PyTorch model across 20 random audio samples of varying durations (1s–8s).

Probability Comparison (sample)

Usage

With ONNX Runtime

import numpy as np
import librosa
import onnxruntime as ort

# Load any variant — just swap the filename
# "model.onnx" (FP32) | "model_fp16.onnx" (FP16) | "model_quantized_matmul_only.onnx" (INT8)
session = ort.InferenceSession("model_fp16.onnx", providers=["CPUExecutionProvider"])

# Load and preprocess audio
audio, sr = librosa.load("audio.wav", sr=16000)

# Normalize (matching Wav2Vec2FeatureExtractor behavior)
audio = (audio - audio.mean()) / np.sqrt(audio.var() + 1e-7)

# Run inference
input_values = audio.reshape(1, -1).astype(np.float32)
attention_mask = np.ones_like(input_values, dtype=np.int64)

logits = session.run(None, {
    "input_values": input_values,
    "attention_mask": attention_mask,
})[0]

# Get probabilities
probs = np.exp(logits) / np.exp(logits).sum(axis=-1, keepdims=True)
real_prob, fake_prob = probs[0]

print(f"Real: {real_prob:.4f}, Fake: {fake_prob:.4f}")
print(f"Verdict: {'Fake/Spoof' if fake_prob > real_prob else 'Real/Bonafide'}")

With Transformers + PyTorch (original model)

import torch
import librosa
from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2ForSequenceClassification

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("Gustking/wav2vec2-large-xlsr-deepfake-audio-classification")
model = Wav2Vec2ForSequenceClassification.from_pretrained("Gustking/wav2vec2-large-xlsr-deepfake-audio-classification")
model.eval()

audio, sr = librosa.load("audio.wav", sr=16000)
inputs = feature_extractor(audio, sampling_rate=16000, return_tensors="pt", padding=True)

with torch.no_grad():
    logits = model(**inputs).logits
    probs = logits.softmax(dim=-1)[0]

print(f"Real: {probs[0]:.4f}, Fake: {probs[1]:.4f}")

Inputs & Outputs

Inputs

Outputs

Conversion Details

• FP32: Exported from PyTorch using the TorchScript ONNX exporter (opset 17) with dynamic axes for variable-length audio input.
• FP16: Weight initializers converted from float32 to float16 with Cast nodes inserted to upcast back to float32 at runtime. This keeps the compute graph in FP32 (full compatibility) while halving storage.
• INT8: Dynamic INT8 quantization applied selectively to MatMul operations only (attention and feed-forward layers). The feature extractor's Conv layers are kept in full precision, as quantizing them introduced significant accuracy degradation (~37% mean relative error vs ~3.66% with MatMul-only).

License

Apache 2.0 (following the original model's license).