hinglish-turn-v2

Hinglish Turn Detection — v2 (s2-004)

Tiny audio-native turn detector for Hinglish (Hindi–English code-mixed) telephony speech: given up to 8 s of audio, is the user done speaking (complete → agent can respond) or just pausing (incomplete → keep listening)? Two-stage fine-tune of the Smart Turn v3.2 approach for the language pair it skipped.

Results

(acc / F1). The intended story: the baseline wins on its own training distribution; the Hinglish fine-tune wins on Hinglish domains. int8 variant: −2.5 / −3.0 (1 clip) / +0.4 acc, 9.05 MB, p50 ~13 ms CPU. Policy (P6 sweep): τ=0.5 default; knee τ≈0.45; test-B flat τ 0.25–0.85. Trailing context: accuracy saturates at ~4 s (test-A) / 2 s (test-B).

Input contract (identical to Smart Turn)

• 16 kHz mono float32 in [-1, 1]
• ≤ 8 s: shorter → zero-pad FRONT; longer → keep the LAST 8 s
• Feature convention: whisper log-mel (80, 800) via chunk_length=8 WITHOUT per-bin normalization (do_normalize=False). The v2 models were trained and exported self-consistently on unnormalized mels (transformers 5 silently dropped the FE default). Do NOT enable normalization at inference.
• Output: 1 logit — apply sigmoid; ≥ 0.5 → complete (at the shipped τ=0.5)

import numpy as np, onnxruntime as ort, soundfile as sf
from huggingface_hub import hf_hub_download
from transformers import WhisperFeatureExtractor

fe = WhisperFeatureExtractor.from_pretrained("openai/whisper-tiny", chunk_length=8)
sess = ort.InferenceSession(hf_hub_download("Shrey160/hinglish-turn-v2", "onnx/s2-004.fp32.onnx"))

audio, sr = sf.read("clip.wav", dtype="float32")   # resample to 16 kHz mono first
feats = fe(audio, sampling_rate=16000, return_tensors="np", padding="max_length",
           max_length=8 * 16000, truncation=True, do_normalize=False)["input_features"]
logit = sess.run(None, {sess.get_inputs()[0].name: feats.astype(np.float32)})[0].reshape(-1)[0]
p_complete = 1 / (1 + np.exp(-logit))

Training (two-stage transfer, whisper-tiny encoder)

1. Stage 1 — frozen encoder, head trained on original Smart Turn hin+eng (7,200 clips), lr 5e-5, 4 epochs.
2. Stage 2 — 232 TTS Hinglish clips (×2 upsample) + 50:50 hin/eng replay, lr 1e-4, 3 epochs, last 2 encoder blocks unfrozen (approved amendment). Ablations (ASP pooling, attention-end, label smoothing, unfreeze k, replay ratio) and KD (whisper-small teacher) all lost to this default or shipped as documented negatives.

Limitations

• test-C is MUCS tutorial monologue — all neural models collapse there (prosody domain gap); treat as stress test, relative rankings only.
• test-B is 33 clips sharing the TTS pipeline with s2 training data.
• mels are unnormalized (see above); normalized retraining = future work.
• 83% of the v3.2 training pool is TTS-generated; human Hinglish data is the top scaling lever.