Downloads · 30 days
75
5% of all-time downloads
nii-yamagishilab/wav2vec-large-anti-deepfake-nda
wav2vec-large-anti-deepfake-nda is a machine learning model from nii-yamagishilab. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as cc-by-nc-sa-4.0.
The AntiDeepfake project provides a series of powerful foundation models post-trained for deepfake detection. The AntiDeepfake model can be used for feature extraction for deepfake detection in a zero-shot manner, or…
Downloads · 30 days
75
5% of all-time downloads
All-time downloads
1.4K
Public
Parameters
317M
1.3 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors1.3 GB · 100%
From the Hugging Face model README
The AntiDeepfake project provides a series of powerful foundation models post-trained for deepfake detection. The AntiDeepfake model can be used for feature extraction for deepfake detection in a zero-shot manner, or it may be further fine-tuned and optimized for a specific database or deepfake-related task.
For technical details and analysis, please refer to our paper Post-training for Deepfake Speech Detection.
All models are released on Hugging Face 🤗 with two variants:
| Model | Variants |
|---|---|
| XLS-R-2B-AntiDeepfake | Default, NDA |
| XLS-R-1B-AntiDeepfake | Default, NDA |
| MMS-1B-AntiDeepfake | Default, NDA |
| MMS-300M-AntiDeepfake | Default, NDA |
| Wav2Vec2-Large-AntiDeepfake | Default, NDA |
| Wav2Vec2-Small-AntiDeepfake | Default, NDA |
| Hubert-Extra-Large-AntiDeepfake | Default, NDA |
Explore training scripts, config files, and evaluation utilities in our GitHub repository:🔗 AntiDeepfake GitHub Repository
facebook/wav2vec2-large-960h-lv60-self 🔗.1e-7.📦 Dependencies:
### New conda environments ###
conda create --name antideepfake python==3.9.0
conda activate antideepfake
conda install pip==24.0
### Install packages ###
pip install huggingface-hub==0.31.1 fairseq==0.12.2 safetensors==0.5.3 soundfile==0.13.1
🚀 Inference:
import os
import torch
import torchaudio
from fairseq.models.wav2vec import Wav2Vec2Model, Wav2Vec2Config
from huggingface_hub import PyTorchModelHubMixin
# This is the only part of the script you need to modify.
# Set this to the path where your audio files are stored.
folder_path = "/path/to/folder/contains/wavs/"
audio_formats = (".mp3", ".wav", ".flac", ".m4a")
# === Set device (use GPU if available) ===
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")
# === Wrapper for the SSL model ===
class SSLModel(torch.nn.Module):
def __init__(self):
super().__init__()
# Model config used to build SSL architecture
cfg = Wav2Vec2Config(
quantize_targets=True,
extractor_mode="layer_norm",
layer_norm_first=True,
final_dim=768,
latent_temp=(2.0, 0.1, 0.999995),
encoder_layerdrop=0.0,
dropout_input=0.0,
dropout_features=0.0,
dropout=0.0,
attention_dropout=0.0,
conv_bias=True,
encoder_layers=24,
encoder_embed_dim=1024,
encoder_ffn_embed_dim=4096,
encoder_attention_heads=16,
feature_grad_mult=1.0
)
# Initialize SSL model with random weights
self.model = Wav2Vec2Model(cfg)
def extract_feat(self, input_data):
# If input has shape (B, T, 1), squeeze the last dim
if input_data.ndim == 3:
input_data = input_data[:, :, 0]
# Extract features
with torch.no_grad():
features = self.model(input_data.to(device), mask=False, features_only=True)['x']
return features
# === Function for reading and pre-processing waveforms ===
def load_wav_and_preprocess(wav_path, target_sr=16000):
# Load audio file
wav, sr = torchaudio.load(wav_path)
# Convert to mono if stereo
wav = wav.mean(dim=0)
# Resample to target sampling rate
wav = torchaudio.functional.resample(wav, sr, new_freq=target_sr)
# Normalize waveform
with torch.no_grad():
wav = torch.nn.functional.layer_norm(wav, wav.shape)
# Add batch dimension and return
return wav.unsqueeze(0).to(device)
# === The actual deepfake detection model using SSL frontend + FC backend ===
class DeepfakeDetector(torch.nn.Module, PyTorchModelHubMixin):
def __init__(self):
super().__init__()
self.ssl_orig_output_dim = 1024
self.num_classes = 2
# Frontend: SSL model
self.m_ssl = SSLModel()
# Backend: Pooling + Classification
self.adap_pool1d = torch.nn.AdaptiveAvgPool1d(output_size=1)
self.proj_fc = torch.nn.Linear(
in_features=self.ssl_orig_output_dim,
out_features=self.num_classes,
)
def forward(self, wav):
emb = self.m_ssl.extract_feat(wav) # [B, T, D]
emb = emb.transpose(1, 2) # [B, D, T]
pooled_emb = self.adap_pool1d(emb) # [B, D, 1]
pooled_emb = pooled_emb.squeeze(-1) # [B, D]
logits = self.proj_fc(pooled_emb) # [B, 2]
return logits
# === Load AntiDeepfake model from Hugging Face===
model = DeepfakeDetector.from_pretrained("nii-yamagishilab/wav2vec-large-anti-deepfake-nda")
model.to(device)
model.eval()
# === Inference on a folder of audio files ===
results = []
for root, _, files in os.walk(folder_path):
for file in files:
if file.lower().endswith(audio_formats):
input_path = os.path.join(root, file)
with torch.no_grad():
wav = load_wav_and_preprocess(input_path)
logits = model(wav)
probs = torch.nn.functional.softmax(logits, dim=1)
results.append((file, probs.cpu().numpy()[0]))
# Sort results alphabetically by filename
results.sort(key=lambda x: x[0])
# Print formatted results
print("\n=== Deepfake Detection Results ===")
for file_name, prob in results:
print(f"{file_name}: real prob = {prob[1]:.3f}, fake prob = {prob[0]:.3f}")
Results shown below can be reproduced using scripts provided in our GitHub repository.
| Test Database | ROC AUC | Accuracy | Precision | Recall | F1-score | FPR | FNR | EER (%) @ Threshold |
|---|---|---|---|---|---|---|---|---|
| ADD2023 | 0.950 | 0.883 | 0.949 | 0.889 | 0.918 | 0.135 | 0.111 | 12.67 @ 0.6192 |
| DeepVoice | 0.990 | 0.807 | 0.389 | 0.994 | 0.559 | 0.219 | 0.006 | 5.04 @ 0.9974 |
| FakeOrReal | 1.000 | 0.970 | 0.998 | 0.940 | 0.968 | 0.002 | 0.060 | 0.80 @ 0.0072 |
| FakeOrReal-norm | 0.999 | 0.986 | 0.980 | 0.991 | 0.986 | 0.019 | 0.009 | 1.45 @ 0.6635 |
| In-the-Wild | 0.997 | 0.977 | 0.988 | 0.975 | 0.981 | 0.021 | 0.025 | 2.25 @ 0.4303 |
| Deepfake-Eval-2024 | 0.773 | 0.730 | 0.734 | 0.921 | 0.817 | 0.630 | 0.080 | 30.06 @ 0.9995 |
You can also fine-tune this model on a specific database, the corresponding code is provided in our GitHub repository. Fine-tuning will follow a similar process to training a new model, except that model weights will be initialized as AntiDeepfake checkpoints.
Below are the evaluation results of this model fine-tuned on the Deepfake-Eval-2024 training set and tested on its corresponding test set (as shown in the previous table):
| Test Input Length | ROC AUC | Accuracy | Precision | Recall | F1-score | FPR | FNR | EER (%) @ Threshold |
|---|---|---|---|---|---|---|---|---|
| 4s | 0.8766 | 0.8086 | 0.8077 | 0.9282 | 0.8638 | 0.4176 | 0.0718 | 20.15 @ 0.9449 |
| 10s | 0.9134 | 0.8704 | 0.8738 | 0.9365 | 0.9041 | 0.2536 | 0.0635 | 12.70 @ 0.9157 |
| 13s | 0.9196 | 0.8857 | 0.8947 | 0.9345 | 0.9142 | 0.2057 | 0.0655 | 11.70 @ 0.8947 |
| 30s | 0.9254 | 0.8964 | 0.9189 | 0.9210 | 0.9200 | 0.1488 | 0.0790 | 10.73 @ 0.7246 |
| 50s | 0.9163 | 0.8798 | 0.9214 | 0.8865 | 0.9036 | 0.1321 | 0.1135 | 12.44 @ 0.5401 |
Below is a breakdown of the training set used for post-training of speech SSL models.
| 📚 Database | 🌍 Language | ✅ Genuine (hrs) | ❌ Fake (hrs) |
|---|---|---|---|
| AISHELL3 | zh | 85.62 | 0 |
| ASVspoof2019-LA | en | 11.85 | 97.80 |
| ASVspoof2021-LA | en | 16.40 | 116.10 |
| ASVspoof2021-DF | en | 20.73 | 487.00 |
| ASVspoof5 | en | 413.49 | 1808.48 |
| CFAD | zh | 171.25 | 224.55 |
| CNCeleb2 | zh | 1084.34 | 0 |
| Codecfake | en, zh | 129.66 | 808.32 |
| CodecFake | en | 0 | 660.92 |
| CVoiceFake | en, fr, de, it, zh | 315.14 | 1561.16 |
| DECRO | en, zh | 35.18 | 102.44 |
| DFADD | en | 41.62 | 66.01 |
| Diffuse or Confuse | en | 0 | 231.66 |
| DiffSSD | en | 0 | 139.73 |
| DSD | en, ja, ko | 100.98 | 60.23 |
| FLEURS | 102 languages | 1388.97 | 0 |
| FLEURS-R | 102 languages | 0 | 1238.83 |
| HABLA | es | 35.56 | 87.83 |
| LibriTTS | en | 585.83 | 0 |
| LibriTTS-R | en | 0 | 583.15 |
| LibriTTS-Vocoded | en | 0 | 2345.14 |
| LJSpeech | en | 23.92 | 0 |
| MLAAD | 38 languages | 0 | 377.96 |
| MLS | 8 languages | 50558.11 | 0 |
| SpoofCeleb | Multilingual | 173.00 | 1916.20 |
| VoiceMOS | en | 0 | 448.44 |
| VoxCeleb2 | Multilingual | 1179.62 | 0 |
| VoxCeleb2-Vocoded | Multilingual | 0 | 4721.46 |
| WaveFake | en, ja | 0 | 198.65 |
| Train Set | Over 100 languages | 56370.00 | 18280.00 |
All AntiDeepfake models were developed by Yamagishi Lab at the National Institute of Informatics (NII), Japan.
All model weights are the intellectual property of NII and are made available for research and educational purposes under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license.
This project is based on results obtained from project JPNP22007, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).
It is also partially supported by the following grants from the Japan Science and Technology Agency (JST):
This study was carried out using the TSUBAME4.0 supercomputer at Institute of Science Tokyo.