AntSpeaker

MECT

<p align="center"> <a href="https://huggingface.co/AntResearch/AntSpeaker"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-AntSpeaker-blue.svg" alt="AntSpeaker on Hugging Face"></a> <a href="https://arxiv.org/abs/2609.24061"><img src="https://img.shields.io/badge/arXiv-2609.24061-b31b1b.svg?logo=arxiv&logoColor=white" alt="arXiv"></a> <a href="https://github.com/ant-research/AntSpeaker"><img src="https://img.shields.io/badge/GitHub-AntSpeaker-181717.svg?logo=github&logoColor=white" alt="GitHub"></a> <img src="https://img.shields.io/badge/license-CC--BY--NC--SA--4.0-lightgrey" alt="License"> </p>

Official model checkpoints for the paper: MECT: Mixture of Experts with CNN-Transformer Network for Speaker verification.

MECT is a speaker verification model that integrates the Mixture-of-Experts (MoE) mechanism into a CNN-Transformer backbone with optimized block structure and stacking scheme. The model explores four MoE variants spanning utterance-level and frame-level granularity with dense and sparse routing strategies, proving that the MoE mechanism is effective over the baseline with only a small increase in parameters.

✨ Key Features

• Mixture of Experts — integrates the MoE mechanism into a CNN-Transformer backbone with optimized block structure and stacking scheme, exploring four MoE variants that span utterance-level and frame-level granularity with dense and sparse routing strategies
• Lightweight and scalable — four model sizes ranging from 3.78M to 9.57M parameters
• Strong performance — competitive EER on VoxCeleb1 benchmarks
• Streaming support — a streaming speaker verification model is also provided for real-time, low-latency scenarios such as live meetings, voice assistants, and on-device applications

📦 Model Variants

This repository provides 4 model sizes with 6 checkpoints in total:

MECT-B2-Causal is a streaming variant of MECT-B2 to support streaming speaker verification.

📊 Performance

VoxCeleb1 Test Sets (EER % / minDCF)

🚀 Quick Start

1. Download Model Checkpoints

From HuggingFace:

# Clone the HuggingFace repo (requires Git LFS)
git lfs install
git clone https://huggingface.co/AntResearch/AntSpeaker

# Or download a single checkpoint
hf download AntResearch/AntSpeaker mect_b2_vc2.pt --local-dir ./

From GitHub:

Model checkpoints are available on HuggingFace. Source code is hosted on GitHub.

2. Installation

pip install torch torchaudio soundfile

3. Extract Speaker Embeddings

import torch
import torchaudio
import soundfile as sf
import torchaudio.compliance.kaldi as kaldi
import antspeaker.models
from antspeaker.utils.registry import create_model

# 1. Load model
ckpt = torch.load("mect_b2_vc2.pt", map_location="cpu")
model = create_model(ckpt["config"]["model"])
model, _ = model.load_model("mect_b2_vc2.pt")
feat_config = ckpt["config"]["feature"]

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
model.eval()

# 2. Load audio (resample to model's sampling rate if needed)
waveform, sr = sf.read("example.wav", dtype="float32")
waveform = torch.from_numpy(waveform.T).unsqueeze(0) 
if sr != feat_config["sampling_rate"]:
    waveform = torchaudio.transforms.Resample(
        orig_freq=sr, new_freq=feat_config["sampling_rate"])(waveform)

# 3. Extract 80-dim Mel filterbank features
fbank = kaldi.fbank(
    waveform * (1 << 15),
    num_mel_bins=feat_config["num_mel_bins"],
    frame_length=feat_config["frame_length"],
    frame_shift=feat_config["frame_shift"],
    dither=feat_config["dither"],
    energy_floor=0.0,
    window_type="hamming",
    sample_frequency=feat_config["sampling_rate"],
)
fbank = fbank.unsqueeze(0).transpose(-1, -2)

# 4. Extract embedding
with torch.no_grad():
    embedding = model(fbank.to(device)).cpu()

4. Speaker Verification

import torch.nn.functional as F

# Extract embeddings for two utterances
emb1 = model(fbank1.to(device)).cpu()
emb2 = model(fbank2.to(device)).cpu()

# Cosine similarity
cosine_sim = F.cosine_similarity(emb1, emb2)
print(f"Cosine similarity: {cosine_sim.item():.4f}")

📝 Citation

If you use MECT in your research, please cite:

@article{zheng2026mect,
    title={MECT: Mixture of Experts with CNN-Transformer Network for Speaker verification}, 
    author={Yu Zheng and Jinghan Peng and ChangHao Zhang and Jian Liu and Weiqiang Wang},
    journal={arXiv preprint arXiv:2609.24061},
    year={2026},
}

📜 License

This model is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC-BY-NC-SA 4.0).

• Non-Commercial — you may not use this model for commercial purposes
• Attribution — you must give appropriate credit
• ShareAlike — derivatives must be licensed under the same terms

See the LICENSE for full details.

🔗 Links

• GitHub Repository: https://github.com/ant-research/AntSpeaker
• HuggingFace Model: https://huggingface.co/AntResearch/AntSpeaker
• Arxiv Paper: https://arxiv.org/abs/2609.24061

🙏 Acknowledgements

We thank the authors of VoxCeleb2 and VoxBlink2 for providing the training corpora.