Audar-Diarization-V1

<div align="center">

Audar-Diarization-V1

Real-time streaming speaker diarization — up to 8 speakers, state of the art on 8 corpora.

From Arabic to the world.

<p><a href="#-what-it-is"><b>🧭 Overview</b></a> · <a href="#-benchmarks"><b>📊 Benchmarks</b></a> · <a href="#-quickstart"><b>⚡ Quickstart</b></a> · <a href="#-real-time-streaming"><b>🎙️ Streaming</b></a> · <a href="#-files"><b>📦 Files</b></a> · <a href="https://github.com/AudarAI/Audar-diarization-V1"><b>🐙 GitHub</b></a> · <a href="https://www.audarai.com"><b>☁️ Audar API</b></a> · <a href="https://www.audarai.com/license/audarai-community-license-v1.0/"><b>📜 License</b></a></p> </div>

---

🧭 What it is

Audar-Diarization-V1 answers "who spoke when" — in real time, for up to 8 speakers, across hour-long multi-speaker audio. It is the speaker-attribution engine of the Audar realtime stack: paired with Audar-ASR-V1 it turns a verbatim transcript into a speaker-labeled one — the difference between an undifferentiated wall of text and a minutes-ready board record.

It is built on NVIDIA's Streaming Sortformer v2.1 and advanced in-house through Audar's diarization program:

• 🔧 Surgical 4→8 speaker head expansion — the released Sortformer supports only 4 speaker slots. Audar extends the output layer to 8 by modifying exactly two Linear layers (adding just 2,312 parameters), cloning the learned 4-speaker weights so the pretrained decision boundary is preserved while capacity opens for speakers 5–8.
• 🧊 Freeze-and-fine-tune — the 109.55M-parameter FastConformer acoustic encoder is frozen; only the lightweight Transformer encoder + Sortformer assignment modules (8.15M) are trained. This buys a 4.2-point DER advantage over full-model fine-tuning and keeps training fast on a single node.
• 🧬 Correct-by-construction synthetic data — Audar fixes a systematic bug in the legacy synthetic- data generator (97 % of samples had labels running past the audio) and generates 200 h of clean 5–8-speaker conversations, on top of 486 h of real far-field meetings.
• 🎯 Arrival-Order Speaker Cache (AOSC) — speakers are assigned to output slots in the order they first speak, resolving the permutation problem without global clustering or Hungarian matching. Identity is held across sessions up to 74 minutes, and the cache auto-sizes to however many speakers are actually present.

The result streams on a single GPU with 1.04 s algorithmic latency and a 0.003 real-time factor (1 s of audio processed in ~3 ms), while posting the lowest DER of any evaluated system on all eight benchmark corpora.

Model summary

<table> <tbody> <tr><td width="220"><b>Model</b></td><td>Audar-Diarization-V1 — streaming speaker diarization (up to 8 speakers)</td></tr> <tr><td><b>Task</b></td><td>Speaker diarization ("who spoke when") — streaming <i>and</i> offline whole-file</td></tr> <tr><td><b>Architecture</b></td><td>Sortformer (encoder-label): frozen FastConformer → trainable Transformer encoder → Sortformer modules + AOSC</td></tr> <tr><td><b>Base</b></td><td>NVIDIA Streaming Sortformer v2.1, surgically extended 4 → 8 speaker slots</td></tr> <tr><td><b>Total parameters</b></td><td>117,696,272 (117.7M)</td></tr> <tr><td><b>Trainable / frozen</b></td><td>8.15M trainable · 109.55M frozen (acoustic encoder)</td></tr> <tr><td><b>Max speakers</b></td><td>8 per session (AOSC auto-sizes to the number present)</td></tr> <tr><td><b>Prediction frame</b></td><td>80 ms</td></tr> <tr><td><b>Algorithmic latency</b></td><td>1.04 s (streaming mode)</td></tr> <tr><td><b>Real-time factor</b></td><td>0.003 (single GPU, batch 1)</td></tr> <tr><td><b>Sample rate</b></td><td>16 kHz mono</td></tr> <tr><td><b>Format</b></td><td>safetensors (fp32, lossless) — PyTorch / CUDA via NeMo</td></tr> <tr><td><b>License</b></td><td>AudarAI Community License v1.0</td></tr> </tbody> </table>

📊 Benchmarks

Evaluated with dscore at a 0.25 s collar, ignoring overlap (DIHARD protocol) on the official dev/eval splits of 8 corpora spanning meetings, dinner parties, broadcast, and in-the-wild audio. Audar-Diarization-V1 posts the lowest DER on every corpus and a macro DER of 22.03 % — beating pyannote 3.1 by 7.63 pp and stock Sortformer v2.1 by 12.16 pp.

DER % per corpus (lower is better)

Where the gain comes from — DER decomposition (macro)

The advantage is confusion: 4.80 % vs 17.2–17.6 % — a 3.6× reduction, from the AOSC's stable identity tracking. The slightly higher false-alarm rate reflects a deliberately assertive streaming VAD (a missed utterance costs more than a brief false activation in live transcription) and is tunable via the onset threshold.

Out-of-domain (CALLHOME, 8 kHz telephony — not in training)

Identity also holds on the longest sessions in the benchmark — e.g. a 74-minute, 5-speaker ICSI meeting at 22.2 % DER with ~2 % confusion.

⚡ Quickstart

Ships as a single fp32 safetensors bundle — model.safetensors + config.yaml + load_diarizer.py. The loader instantiates the NeMo Sortformer model and loads the weights directly (no .nemo tar):

# needs: nemo_toolkit[asr]>=2.6, safetensors, omegaconf
from huggingface_hub import snapshot_download
import sys; sys.path.insert(0, snapshot_download("audarai/Audar-Diarization-V1"))
from load_diarizer import load_diarizer

model = load_diarizer()                                   # fp32, CUDA (device="cpu" also works)
segs  = model.diarize(audio=["meeting.wav"], batch_size=1)
# → RTTM-style [(start_s, end_s, speaker_slot), ...] per file

Runnable inference examples (offline diarization + RTTM export), the full 8-corpus benchmark, and reproduction steps are open at github.com/AudarAI/Audar-diarization-V1.

🎙️ Real-time streaming

The same checkpoint runs true streaming via forward_streaming_step with persistent spkcache / FIFO state: ~1 s chunks, 80 ms prediction frames, up to 8 concurrent speakers, and session-stable slot labels that never rewrite once committed. Algorithmic latency is 1.04 s at a 0.003 real-time factor on a single GPU.

Speaker-attributed transcription. The Audar serving gateway runs diarization in parallel with Audar-ASR-V1 and assigns each transcribed word to the speaker dominant during its time span — so combined latency is the max of the two streams, not the sum. One deployment exposes ASR-only, diarization-only, and ASR+diarization endpoints over HTTP and an OpenAI-Realtime-compatible WebSocket. For a managed, production-hosted endpoint, see the Audar API.

📦 Files

Lossless fp32, and faster to load. The model.safetensors carries the full-precision weights bit-for-bit — verified by a round-trip check (990/990 tensors identical) and by downcasting to the prior fp16 release with zero mismatches across all 971 float tensors, so it reproduces the exact model. The weight-load step is ~28× faster than the legacy .nemo (≈12 ms mmap vs ≈344 ms untar + unpickle), and safetensors is the safe, community-standard format (no arbitrary-code pickle path).

<details> <summary><b>Re-quantizing to fp16 (optional)</b></summary>

load_diarizer.py auto-detects weight dtype, so you can quantize model.safetensors to fp16 yourself and it will load unchanged. When fp16 is detected the loader keeps the preprocessor (STFT/mel) in fp32 and runs the streaming path under torch.set_default_dtype(torch.float16) (NeMo's streaming state is otherwise created dtype-less). ONNX export is not supported out-of-the-box (NeMo 2.6.2's Sortformer export needs streaming-state wiring).

</details>

Intended use & limitations

Intended use. Speaker-attributed meeting/broadcast/call-center transcription, board and panel recordings, and any real-time or offline "who spoke when" task — cloud, on-prem, or edge.

Limitations.

• Up to 8 speakers per session; very large panels beyond 8 are out of scope.
• Far-field, high-overlap, extreme-noise audio (e.g. CHiME-6-style dinner parties) remains the hardest case for every system.
• Very-low-activity speakers (<2 % of talk time) can have their cache entry decay during long silences.
• Not evaluated for, and must not be used for, covert speaker identification.

📜 License

Released under the AudarAI Community License v1.0 — research and limited commercial use for qualifying Community Entities; enterprise, large-scale, or model-as-a-service use requires an AudarAI Enterprise License. See audarai.com/license/audarai-community-license-v1.0, or contact [email protected] for enterprise licensing.

Citation

@techreport{audar-diarization-v1-2026,
  title       = {Audar-Diarization-V1: Real-Time Streaming Speaker Diarization for Long-Form, Multi-Speaker Audio},
  author      = {Audar AI Team},
  institution = {AudarAI},
  year        = {2026},
  url         = {https://huggingface.co/audarai/Audar-Diarization-V1}
}

---

About AudarAI

<div align="center">

Leading Arabic-First Multilingual Audio Intelligence

AudarAI starts with Arabic — and expands to the world.

</div>

We are building advanced multilingual audio intelligence that helps individuals, enterprises, and governments communicate across languages, cultures, and borders. By combining Arabic-first speech technology with global multilingual AI, AudarAI transforms voice into understanding, interaction, and connection.

Our work spans speech recognition, speech understanding, speaker diarization, voice-enabled digital assistants, human-computer interaction, and intelligent audio systems designed for real-world impact. From empowering people to access technology in their native language to helping organizations communicate globally, AudarAI is shaping a future where every voice can be heard, understood, and connected.

Arabic-first. Multilingual by design. Human-centered at heart.

<div align="center">

🌐 www.audarai.com · 🤗 Hugging Face · GitHub · [email protected]

© 2026 AUDARAI PTE. LTD. · Licensed under the AudarAI Community License v1.0

</div>