turn-detector-v2

turn-detector-v2 - Turkish Turn Detection Model

<p align="center"> <a href="https://huggingface.co/hayatiali/turn-detector-v2"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-turn--detector--v2-yellow" alt="Hugging Face"></a> <a href="https://huggingface.co/hayatiali/turn-detector-v2"><img src="https://img.shields.io/badge/Model-Production%20Ready-brightgreen" alt="Production Ready"></a> <img src="https://img.shields.io/badge/Language-Turkish-blue" alt="Turkish"> <img src="https://img.shields.io/badge/Task-Turn%20Detection-orange" alt="Turn Detection"> <img src="https://img.shields.io/badge/F1-97.69%25-success" alt="F1 Score"> </p>

This model is designed for detecting turn-taking patterns in Turkish conversations, optimizing voice assistant latency by identifying when user utterances require LLM processing vs. simple acknowledgments.

Developed by SiriusAI Tech Brain Team

---

Mission

To optimize voice assistant response latency by detecting when user utterances require LLM processing vs. simple acknowledgments.

The turn-detector-v2 model analyzes conversational turn pairs (bot utterance + user response) and classifies whether the user's response requires LLM processing (agent_response) or is just a backchannel acknowledgment that can be handled without LLM (backchannel).

Key Benefits

---

Model Overview

---

Performance Metrics

Final Evaluation Results

Per-Class Performance

---

Semantic Classification Rules

When to Classify as backchannel (Skip LLM)

When to Classify as agent_response (Send to LLM)

Golden Rule

If bot asked a question → Always agent_response
If bot gave info + User short acknowledgment → backchannel

---

Dataset

Dataset Statistics

Label Distribution

Domain Coverage

• E-commerce (kargo, iade, teslimat)
• Banking (hesap, bakiye, kredi)
• Telecom (numara tasima, data, hat)
• Insurance (prim, police, teminat, kasko)
• General Support (sikayet, yonetici, eskalasyon)
• Identity Verification (TC, gorusuyorum, soyadi)

---

Label Definitions

Input Format

[bot] <bot utterance> [sep] [user] <user response>

Example Classifications

agent_response (Send to LLM):

[bot] size nasil yardimci olabilirim [sep] [user] fatura sorgulamak istiyorum
[bot] ahmet bey ile mi gorusuyorum [sep] [user] evet benim
[bot] islemi onayliyor musunuz [sep] [user] evet onayliyorum
[bot] kargonuz yolda [sep] [user] ne zaman gelir
[bot] poliçeniz aktif [sep] [user] teminat limitini ogrenebilir miyim

backchannel (Skip LLM, return filler):

[bot] faturaniz 150 tl gorunuyor [sep] [user] tamam
[bot] siparisiniz 3 gun icinde teslim edilecek [sep] [user] anladim
[bot] kaydinizi kontrol ediyorum [sep] [user] peki
[bot] policeniz yenilendi [sep] [user] tesekkurler
[bot] sifreni sms ile gonderdik [sep] [user] ok aldim

---

Training

Hyperparameters

---

Usage

Installation

pip install transformers torch

Quick Start

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "hayatiali/turn-detector-v2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

LABELS = ["agent_response", "backchannel"]

def predict(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
    with torch.no_grad():
        outputs = model(**inputs)
        probs = torch.softmax(outputs.logits, dim=-1)[0]

    scores = {label: float(prob) for label, prob in zip(LABELS, probs)}
    return {"label": max(scores, key=scores.get), "confidence": max(scores.values())}

# Bot asks question → agent_response
print(predict("[bot] ahmet bey ile mi gorusuyorum [sep] [user] evet benim"))
# Output: {'label': 'agent_response', 'confidence': 0.99}

# Bot gives info + User acknowledges → backchannel
print(predict("[bot] faturaniz 150 tl gorunuyor [sep] [user] tamam"))
# Output: {'label': 'backchannel', 'confidence': 0.98}

Production Integration

class TurnDetector:
    """Production-ready turn detection for voice assistants."""

    LABELS = ["agent_response", "backchannel"]
    FILLER_RESPONSES = ["hmm", "evet", "tamam", "anlıyorum"]

    def __init__(self, model_path="hayatiali/turn-detector-v2"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_path)
        self.model = AutoModelForSequenceClassification.from_pretrained(model_path)
        self.device = "cuda" if torch.cuda.is_available() else "cpu"
        self.model.to(self.device).eval()

    def should_call_llm(self, bot_text: str, user_text: str) -> dict:
        """
        Determines if user response should go to LLM.

        Returns:
            dict with 'call_llm' (bool), 'label', 'confidence', 'filler' (if backchannel)
        """
        text = f"[bot] {bot_text} [sep] [user] {user_text}"
        inputs = self.tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
        inputs = {k: v.to(self.device) for k, v in inputs.items()}

        with torch.no_grad():
            probs = torch.softmax(self.model(**inputs).logits, dim=-1)[0].cpu()

        label_idx = probs.argmax().item()
        label = self.LABELS[label_idx]
        confidence = probs[label_idx].item()

        result = {
            "call_llm": label == "agent_response",
            "label": label,
            "confidence": confidence
        }

        if label == "backchannel":
            import random
            result["filler"] = random.choice(self.FILLER_RESPONSES)

        return result

# Usage
detector = TurnDetector()

# Case 1: Bot asks, user confirms → Send to LLM
result = detector.should_call_llm("siparis iptal etmek ister misiniz", "evet iptal et")
# {'call_llm': True, 'label': 'agent_response', 'confidence': 0.99}

# Case 2: Bot informs, user acknowledges → Return filler
result = detector.should_call_llm("siparisiz yola cikti", "tamam")
# {'call_llm': False, 'label': 'backchannel', 'confidence': 0.97, 'filler': 'hmm'}

---

Limitations

---

Citation

@misc{turn-detector-v2-2025,
  title={turn-detector-v2: Turkish Turn Detection for Voice Assistants},
  author={SiriusAI Tech Brain Team},
  year={2025},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/hayatiali/turn-detector-v2}},
  note={Fine-tuned from dbmdz/bert-base-turkish-uncased}
}

---

Contact

• Developer: SiriusAI Tech Brain Team
• Email: [email protected]
• Repository: GitHub

---

Changelog

v2.0 (Current)

Semantic Rule Improvements:

• If bot asks a question → always agent_response (731 rows corrected)
• Rhetorical questions ("really?", "is that so?") → remain as backchannel
• If user asks a real question ("when?", "how?") → agent_response

Dataset Expansion (+9,082 samples):

Metrics: Macro F1: 0.9769, Accuracy: 97.94%

Note: Metrics appear slightly lower than v1.0, but this is a more accurate model. v1.0 had mislabeled data (bot asked question + "yes" = backchannel), which the model memorized. v2.0 ensures semantic consistency.

v1.0

• Initial release
• Dataset: 56,228 samples
• Macro F1: 0.9924, Accuracy: 99.3%

---

License: SiriusAI Tech Premium License v1.0

Commercial Use: Requires Premium License. Contact: [email protected]