conformer-voice-turn-taking

Conformer Voice Turn-Taking & Barge-in Detector

A lightweight, end-to-end Conformer-based acoustic classification model (~2.1M parameters) engineered for low-latency conversational AI agents, real-time voice bots, and turn-taking pipelines.

---

Overview

Traditional Voice Activity Detection (VAD) relies heavily on arbitrary silence timeouts (500–800 ms), leading to awkward dead air or unintended user interruptions.

This model evaluates acoustic prosody, pitch contour, and speech dynamics over a 1.5-second sliding window to classify conversational intent in real time, delivering sub-15ms inference latency on commodity CPUs using ONNX Runtime.

---

Classes

---

Model Specifications

• Architecture: Conformer-based EncDecClassificationModel (~2.1M parameters)
• Audio Input: 16 kHz Mono PCM
• Window Size: 1.5 seconds (24,000 samples)
• Sliding Step: 200 ms (3,200 samples)
• Runtime Target: ONNX Runtime (<15 ms CPU latency)
• Artifacts:
  • turn_taking_final.onnx: Optimized ONNX computation graph for production inference.
  • turn_taking_final.nemo: NVIDIA NeMo checkpoint containing network architecture and PyTorch weights.
  • turn_taking_model.yaml: Feature extractor and model hyperparameters configuration.

---

Quickstart

ONNX Runtime Inference

import numpy as np
import onnxruntime as ort

# Load ONNX session
session = ort.InferenceSession("turn_taking_final.onnx")

# Dummy 1.5s audio slice (16 kHz mono)
audio_chunk = np.random.randn(1, 24000).astype(np.float32)
audio_length = np.array([24000], dtype=np.int64)

# Run inference
inputs = {
    session.get_inputs()[0].name: audio_chunk,
    session.get_inputs()[1].name: audio_length,
}
logits = session.run(None, inputs)[0]

# Compute softmax probabilities
probs = np.exp(logits) / np.sum(np.exp(logits), axis=-1, keepdims=True)
labels = ["complete", "incomplete", "barge_in"]

result = dict(zip(labels, probs[0]))
print(result)