dualturn-endpointing

DualTurn Endpointing

A small, causal, audio-only turn-taking model for voice agents. It listens to both channels of a live conversation (user + agent) and emits, frame-by-frame at 12.5 Hz, the signals a voice agent needs to decide when to speak — with end-of-turn (endpointing) as the primary output.

Adapted from the DualTurn paper (arXiv:2603.08216) and fine-tuned for the endpointing head. It runs on frozen Mimi features, adds only ~1.5M parameters on top of the encoder, needs no ASR and no cloud, and is designed to run on-device / on CPU.

How it's meant to be used

The model sits inside a live two-channel call between a user (CH0) and the agent (CH1). It continuously listens to both channels at the same time and, every 80 ms, emits per-frame predictions: VAD and FVAD for both speakers, and the user's end-of-turn (EOT). A voice agent consumes these — chiefly the user's EOT — to decide when it's the agent's turn to speak (and VAD/FVAD to sense who's talking and who's about to talk). The agent channel is not just context: hearing the agent's own speech (and overlaps/interruptions) is what makes the user-EOT judgment reliable in a real conversation.

It can also run as a single-stream (user-only) detector: feed the user audio and silence the agent channel (pass mono, or a zeros agent channel). You still get the user's EOT + user VAD/FVAD; you simply lose the agent-context benefit. This is the setup used for benchmarking and for user-mic-only deployments.

Frames are 80 ms (12.5 Hz); T ≈ audio_seconds × 12.5. Streaming/causal — outputs at time t use only audio up to t.

Inference modes & latency

Four ready-to-use paths (all fp32, same weights). Offline processes a whole file at once; streaming runs per 80 ms tick for live calls. ONNX paths need only onnxruntime (no PyTorch).

Offline latency is amortized batch throughput (all frames returned after the file is processed); streaming latency is the real per-tick cost in a live call — both well under the 80 ms budget. Streaming is causal + bounded (transformer KV capped at 250 frames = 10 s), so per-tick latency stays flat over arbitrarily long calls. Streaming outputs are decision-equivalent to offline — they match the batch model to within ~0.02–0.03 (fp32 op-order drift in the incremental path), which doesn't move threshold+sustain endpoint decisions.

Inference interval (compute ↔ latency)

Two streaming ONNX graphs are shipped — pick by how often you need to react vs how much CPU you want to spend. Both are strictly causal (the transformer runs frame-by-frame inside the graph — no future audio) and decision-equivalent to offline — they track the batch model to within ~0.02–0.03 (fp32 op-order drift in the streaming path; the two graphs are bit-identical to each other, Δ≈0). The 240 ms graph runs one shared encoder pass over 3 frames instead of three separate passes, so it's ~1.8× cheaper per second of audio:

• stream_tick.onnx — 1 frame per call ([2, 1920] = 80 ms).
• stream_tick_240.onnx — 3 frames per call ([2, 5760] = 240 ms), returns all 3 × 80 ms frames.

Measured on a real call (CPU, 4 threads, carried KV cache). Grace = idle time left in the interval after inference (interval − latency); CPU busy = per-stream duty cycle, fraction of wall-clock the inference runs to keep up with one live stream (latency × calls/sec):

• 80 ms — snappiest, but heaviest. Each call eats 52 ms → only 28 ms grace, pool busy ~65% per stream. Fastest reactions, least room for anything else.
• 240 ms — much cheaper, lots of slack. Each call takes 85 ms (vs 3 × 52 = 156 ms for three 80 ms ticks) but leaves 155 ms grace and sits at ~36% busy.
• The real win is at scale. CPU per second of audio drops ~1.8× (650 → 354 ms/s, ≈45% less), so one machine holds roughly ~1.8× as many concurrent calls — the per-stream % looks modest, but that factor multiplies across every simultaneous call in your fleet. (CPU busy % is a single-stream duty cycle measured with 4 threads — read it as relative cost between graphs, not absolute single-core load.)
• Accuracy is unaffected — the 240 ms graph is decision-equivalent to the 80 ms one (endpoint decisions match); the only cost is that you learn the turn ended at the next 240 ms boundary (up to one interval later).

1. Offline — PyTorch

import torch, torchaudio
from transformers import AutoModel

model = AutoModel.from_pretrained("anyreach-ai/dualturn-endpointing", trust_remote_code=True).eval()

wav, sr = torchaudio.load("conversation.wav")   # [2, T]  CH0 = user, CH1 = agent
with torch.no_grad():
    out = model(wav, sr=sr)

out.eot_probs    # [1, T, 1]      P(user end-of-turn)
out.vad_probs    # [1, T, 2]      P(speaking) — (user, agent)
out.fvad_probs   # [1, T, 2, 4]   P(speaks soon) — (user, agent) × (0-240/240-640/640-1200/1200-2000 ms)

Turn the end-of-turn stream into a decision with a thin policy — e.g. fire when eot_probs stays above a threshold for a short sustain window:

eot = out.eot_probs[0, :, 0]
thr, sustain = 0.5, 3                       # 3 frames ≈ 240 ms
fired = (eot > thr).unfold(0, sustain, 1).all(-1).nonzero()   # frame indices of detected turn-ends

Mono input [T] or [1, T] is also accepted (treated as user with a silent agent — a single-stream fallback). Any sample rate works (sr=); audio is internally resampled to 24 kHz (Mimi's input rate). The call above is the offline mode (whole file at once). For live calls, use the streaming mode below.

2. Streaming — PyTorch (real-time)

For live calls, don't re-run the whole file each tick — use the stateful streamer. It's causal (uses only past audio), keeps bounded state (transformer KV window + LSTM state), and runs ~63 ms per 80 ms tick on CPU (≥4 threads), flat regardless of call length.

from transformers import AutoModel
model = AutoModel.from_pretrained("anyreach-ai/dualturn-endpointing", trust_remote_code=True).eval()
streamer = model.streaming()

# feed audio as it arrives, in 80 ms chunks: [2, 1920] @ 24 kHz  (CH0=user, CH1=agent)
for chunk in chunks_at_24kHz:            # mono [1920] is also accepted (silent agent)
    out = streamer.push(chunk)           # None until the next 12.5 Hz frame is ready
    if out is not None:
        p_eot = out.eot_probs[0, -1, 0].item()      # latest P(user end-of-turn)
        # fire your end-of-turn policy on p_eot (threshold + short sustain)

streamer.reset()                         # between calls

push returns a DualTurnOutput (same eot_probs / vad_probs / fvad_probs) for the new frame(s). It matches the offline model to ~3e-4 mean H1. Feed 24 kHz (resample the continuous stream once, not per-chunk, to avoid chunk-edge artifacts).

3. Offline — ONNX (no PyTorch)

A self-contained model.onnx (Mimi encoder + readout in one graph) is included for CPU / on-device deployment with just onnxruntime. Input is a 2-channel waveform at 24 kHz [2, T] (CH0=user, CH1=agent); resample first with any tool.

import numpy as np, librosa, onnxruntime as ort
from huggingface_hub import hf_hub_download

onnx_path = hf_hub_download("anyreach-ai/dualturn-endpointing", "model.onnx")
sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])

wav, sr = librosa.load("conversation.wav", sr=24000, mono=False)   # [2, T] @ 24 kHz, CH0=user CH1=agent
eot, vad, fvad = sess.run(None, {"audio_24k": wav.astype("float32")})
# eot  [1, T', 1]      P(user end-of-turn)
# vad  [1, T', 2]      P(speaking) — (user, agent)
# fvad [1, T', 2, 4]   P(speaks soon) — (user, agent) × (0-240/240-640/640-1200/1200-2000 ms)

eot = eot[0, :, 0]
fired = np.where(np.convolve((eot > 0.5).astype(int), np.ones(3), "valid") == 3)[0]   # sustained ≥240 ms

Runs comfortably faster than real-time on CPU (≈1 s of compute per 10 s of audio, single-threaded). The graph is fully self-contained — no torch or transformers needed at inference.

4. Streaming — ONNX (real-time, no PyTorch)

Torch-free real-time streaming via stream_tick.onnx + the onnx_streaming.py helper (DualTurnONNXStreamer). Each 80 ms tick only computes the new frame — the graph carries all state (conv buffer, transformer KV window, downsample history, LSTM state) in/out, so latency is low (~52 ms/tick) and flat over long calls. Decision-equivalent to the PyTorch model.

from onnx_streaming import DualTurnONNXStreamer     # download onnx_streaming.py + stream_tick.onnx from the repo

streamer = DualTurnONNXStreamer()                    # auto-downloads stream_tick.onnx (via huggingface_hub)
for chunk in chunks_24k:                             # each: [2, 1920] float32 @ 24 kHz (CH0=user, CH1=agent; mono OK)
    out = streamer.push(chunk)
    p_eot = out["eot"]                               # P(user end-of-turn); also out["vad"] [2], out["fvad"] [2,4]
    # fire your end-of-turn policy on p_eot (threshold + short sustain)
streamer.reset()                                     # between calls

Needs only onnxruntime + numpy (+ huggingface_hub for the auto-download). Feed 24 kHz audio in exactly 1920-sample (80 ms) chunks.

To spend ~1.8× less CPU (at the cost of reacting every 240 ms instead of 80 ms), use the 240 ms graph stream_tick_240.onnx via DualTurnONNXStreamer240 — same helper file, [2, 5760] chunks, returns the 3 × 80 ms frames per call. Both graphs are strictly causal and decision-equivalent to offline; see Inference interval.

from onnx_streaming import DualTurnONNXStreamer240      # downloads stream_tick_240.onnx

streamer = DualTurnONNXStreamer240()
for chunk in chunks_240ms:                               # each: [2, 5760] float32 @ 24 kHz
    for fr in streamer.push(chunk):                      # list of 3 frames: fr["eot"], fr["vad"], fr["fvad"]
        ...                                              # apply your end-of-turn policy on fr["eot"]
streamer.reset()

Performance

Benchmarked on LiveKit's eot-bench (a neutral end-of-turn benchmark). Metric is a latency / false-cutoff Pareto: FC@N ms = % of mid-turn pauses falsely fired on at an N-ms latency budget (lower is better); Lat@X% = ms of dead-air after a true turn-end at an X% false-cutoff budget (lower is better). Our harness reproduces LiveKit's published SmartTurn numbers exactly, so rows are directly comparable; DualTurn is scored single-stream (agent = silence) — an honest handicap. English slice:

DualTurn Endpointing is the best on-device / audio-only turn detector on the board — beating LiveKit's own on-device v1-mini, SmartTurn v3.2, and the open 8B ultraVAD, at ~1.5M params without transcription or cloud. Only closed cloud end-of-turn services lead. It holds the same #1-on-device rank on zero-shot Spanish (not trained on Spanish), evidence that turn-taking here is language-agnostic acoustics.

How it works

Adapted from the DualTurn paper (arXiv:2603.08216) and fine-tuned for endpointing: both channels are encoded by a frozen Mimi speech encoder into 12.5 Hz continuous features; a small causal readout (~1.5M params) produces the per-frame turn-taking probabilities above. Audio-only (no ASR), streaming, and light enough for CPU / on-device inference. The Mimi encoder is downloaded automatically from kyutai/mimi on first use.

Training & evaluation

• Method. Built on the approach introduced in the DualTurn paper (frozen dual-channel Mimi features → small causal turn-taking readout), specialized here for user end-of-turn endpointing.
• Training data. The two public dual-channel (human–human) corpora used in the DualTurn paper — Switchboard and OtoSpeech — plus a large private dual-channel dataset of real AI-agent phone calls (the production domain this model targets).
• Evaluation. Measured on our internal private test set (held-out real agent calls; the recall / interval and latency numbers above) and, for an external neutral comparison, on LiveKit's eot-bench (reported in Performance above; other systems' rows are LiveKit's published numbers).

Files

• model.safetensors — the ~1.5M-param readout + input-standardization stats (PyTorch paths).
• config.json — with auto_map for trust_remote_code.
• modeling_dualturn.py — PyTorch model + both PyTorch modes: offline (model(wav, sr)) and streaming (model.streaming()).
• model.onnx — self-contained offline ONNX graph (Mimi encoder + readout).
• stream_tick.onnx — self-contained streaming ONNX graph (one 80 ms tick, state in/out).
• stream_tick_240.onnx — 240 ms streaming graph (3 frames/tick, ~1.8× cheaper/sec of audio, strictly causal).
• onnx_streaming.py — DualTurnONNXStreamer (80 ms) + DualTurnONNXStreamer240 (240 ms) helpers (pure onnxruntime + numpy).

Notes & limitations

• This is a perception model (it emits turn-taking signals, not agent actions) — pair it with a thin decision policy.
• eot_probs is the user's end-of-turn (this is an endpointing model). vad_probs and fvad_probs cover both channels (user, agent).
• Best in conversational voice-agent (telephone) audio, the domain it was trained on.
• Deps by path: PyTorch modes (1, 2) need torch + torchaudio + transformers; ONNX modes (3, 4) need only onnxruntime + numpy (+ librosa/soundfile for loading, huggingface_hub for download).

Author

Shangeth Rajaa — Senior ML Research Scientist, Anyreach AI.

Citation

@misc{rajaa2026dualturnlearningturntakingdualchannel,
      title={DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining},
      author={Shangeth Rajaa},
      year={2026},
      eprint={2603.08216},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2603.08216},
}