nemotron-3-diarization-coreml

Nemotron 3 Diarization — CoreML for Apple Silicon

CoreML conversion of NVIDIA's Nemotron 3 Diarization (8-speaker streaming Sortformer, 100M params, 31-layer RoPE Transformer, 10 ms output resolution), optimized for on-device inference on Apple platforms via FluidAudio.

Runs fully on-device — Neural Engine or GPU — on macOS 14+ / iOS 17+.

Highlights (M5 Pro)

• Reproduces NVIDIA's model-card accuracy on-device: AMI MHM offline 9.47 DER (card 9.25), low 9.75 (card 9.48) — same forced-alignment references, collar 0. The ~0.25 gap is the fp16 CoreML runtime against NVIDIA's bf16 GPU numbers, and it is consistent across presets rather than drifting.
• fast128 is the accuracy pick: 9.36 DER at 546x wall real-time, and it counts speakers correctly on all 16 AMI test meetings — better than any row on NVIDIA's card, which tops out at 87.5% speaker-counting accuracy.
• 8-speaker streaming down to 1.04 s latency, the card's own reference config.
• 100% ANE-resident W8A8 split builds at half the weight footprint: c128-split-w8a8 gives up 0.27 DER for 95 MB and keeps 16/16 speaker counting.

Which model should I use?

¹ AMI MHM test, 16 meetings, forced-alignment references (nttcslab-sp/diar-forced-alignment), collar 0, overlap included — the protocol NVIDIA's card uses. SCA = speaker-counting accuracy (fraction of meetings whose speaker count is exactly right). NVIDIA's published AMI MHM numbers under this protocol: 9.25 DER / 87.5 SCA offline, 9.48 / 81.25 at 1.04 s. Wall RTFx is single-stream on an M5 Pro MacBook, ANE route for the split builds.

Window size drives speaker counting, not quantization. On the 10.24 s window the W8A8 build matches fp16 at 16/16 meetings for 0.27 DER. On the 2.56 s window it loses 18.8 points of counting accuracy. Short windows are the fragile ones in fp16 too — low and fast sit at 75.0 and 68.8 on a 0.72 s window. If speaker count matters more than latency, prefer a longer window.

Presets

All presets share one checkpoint; they differ only in streaming shape. Latency = (chunk + right-context) x 80 ms. Packed T is the transformer sequence length (speaker cache + FIFO + window) and is what sets cost per call. DER/SCA: AMI MHM test, 16 meetings, forced-alignment references, collar 0.

Monolithic (single model file, simplest integration)

low and fast run the same 0.72 s window at the same latency; the only difference is the FIFO (264 vs 40), which takes the sequence from 317 to 541. That buys low 0.32 DER and 6.2 points of speaker counting, and costs roughly a third of the throughput.

Other configurations (sub-second latency tiers, intermediate chunk sizes, weight-only int8) were built and benchmarked but are not published — they are dominated by the presets above. Open an issue if you need one; regeneration is scripted.

Split-graph (host pre-encode; 100% ANE-resident CoreML subgraph)

Feature stacking, the 1024->512 projection (pre_encode_proj_t.bin), state packing, and masks run host-side; the model is the pure-fp transformer+head. FluidAudio handles all of this via the -split presets.

Usage (FluidAudio, Swift)

import FluidAudio

let config = Nemotron3Config.fast32  // or .preset(named: "c128-split-w8a8")!
let models = try await Nemotron3Models.load(
    config: config,
    directory: modelsDirectoryURL  // this repo's files
)
let diarizer = Nemotron3Diarizer(config: config, models: models)

let (probs, frames) = try diarizer.processComplete(audioSamples)  // 16 kHz mono
let segments = Nemotron3Diarizer.segments(probabilities: probs, frameCount: frames)
// -> arrival-ordered speaker segments at 10 ms resolution, up to 8 speakers

Optional VAD gating for sparse audio (2x wall speedup at 55% speech density):

let (probs, frames) = try diarizer.processComplete(audioSamples, speechMask: mask)

Files

• monolithic/ — one .mlmodelc per preset (mel features in, predictions out)
• monolithic/v2/ — same presets re-exported with fp16 inputs/outputs and gather-free state packing, so they compile for the ANE on M3-class chips (FluidAudio #951); same checkpoint and weights, DER-neutral on AMI. FluidAudio v0.17.3+ loads these; the originals in monolithic/ are kept for older releases.
• split/ — W8A8 transformer graphs (packed/attn_bias/output_mask inputs)
• learnable_sil_emb.bin — learned silence embedding, 512 x fp32 (cache compression)
• pre_encode_proj_t.bin — FeatureStacking projection W^T, 1024x512 fp32 (split mode)

All models: fixed shapes, fp16 weights unless noted, minimum_deployment_target iOS 17 / macOS 14. Conversion verified against the NeMo reference across four state configurations per preset (cold start, half-full cache, steady state, partial final chunk): the export wrapper is exact in torch (0.0), and the CoreML fp16 runtime stays within 1.9e-4 on speaker predictions.

Attribution & license

Converted from nvidia/Nemotron-3-Diarization (Nemotron-3-Diarization.nemo, sha256 867c53f5…), released by NVIDIA under the OpenMDW License v1.1, which permits commercial use. The converted weights carry the same licence. Conversion pipeline, Swift runtime, and benchmarks by Fluid Inference.

Reference: Streaming Sortformer.