livekit-turn-detector-fisher-eot-lora

LiveKit Turn-Detector — Fisher LoRA

LoRA adapter on top of livekit/turn-detector (Qwen2.5-0.5B), fine-tuned on the Fisher English telephone corpus (LDC2004T19 + LDC2005T19) for end-of-turn (EOT) detection.

The pretrained LiveKit detector is strong on structured voice-assistant inputs but weaker on natural conversational speech. This adapter recovers ground on telephone-style natural conversation while leaving the base weights untouched.

Results

Evaluated on a 1,000-example balanced Fisher validation split (split details in data/fisher_splits.json of the source repo):

ΔPR-AUC: +0.068 over the pretrained baseline on Fisher.

Cross-domain transfer (SWDA / AMI / Taskmaster / fixie-ai) is documented in the source repo. The LoRA targets Fisher-style natural telephone speech; expect smaller or zero gains on structured voice-assistant inputs.

Inference

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_id = "livekit/turn-detector"
lora_id = "tuanamz/livekit-turn-detector-fisher-eot-lora"

tok = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.float32)
model = PeftModel.from_pretrained(base, lora_id).eval()

messages = [
    {"role": "user", "content": "yeah i think that was it"},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
if text.rstrip().endswith("<|im_end|>"):
    text = text.rstrip()[: -len("<|im_end|>")]
enc = tok(text, return_tensors="pt", truncation=True, max_length=512)

eou_id = tok.encode("<|im_end|>", add_special_tokens=False)[0]
with torch.no_grad():
    logits = model(**enc).logits[0, -1, :]
prob_eot = float(torch.softmax(logits, dim=-1)[eou_id])
print(f"P(end-of-turn) = {prob_eot:.3f}")

Training details

• Loss: BCE-with-logits on z = logit[<|im_end|>] − logsumexp(other logits). sigmoid(z) is the EOT score.
• Trainable parameters: 1.84 M (1.35% of 136 M) — LoRA on {q,k,v,o}_proj, r=16, α=32, dropout=0.05.
• Optimizer: AdamW, cosine schedule, peak LR 2e-4.
• Batch: 32, single B300 GPU.
• Epochs: 3 (best checkpoint by Fisher-val PR-AUC, saved every 500 steps).
• Training data: 10,529 Fisher conversations → ~1.54 M (positive, partial-truncated-negative) example pairs.

Pickling val_loss as the model-selection metric collapses the model toward the binary class prior; PR-AUC tracking is essential. Full audit in the source repo's doc/phase2_summary.md.

Intended use

End-of-turn detection for voice assistants and full-duplex robot dialogue systems on English lowercase ASR transcripts of natural conversational speech (telephone, casual). Output is P(<|im_end|>); threshold around the model's calibration sweet spot (typically 0.14–0.5 depending on FP/FN cost).

Limitations

• English only. Fisher is American English telephone speech; performance on other dialects, accents, or non-conversational domains is not characterized here.
• Text-only. Prosodic cues (pitch, energy, pause length) are not used — the deployment ceiling on natural speech with text alone is fundamentally bounded.
• The training labels are synthetic: positives are real turn-end utterances; negatives are first-60% truncations of those same utterances. Real ASR partials are different.

Citation

If you use this adapter, please cite the base model:

@misc{livekit-turn-detector,
  title  = {LiveKit Turn-Detector},
  author = {LiveKit},
  url    = {https://huggingface.co/livekit/turn-detector}
}

and the Fisher corpus (LDC2004T19, LDC2005T19).

License

Inherits from livekit/turn-detector. Fisher corpus terms apply to anything trained on it.

Framework versions

• PEFT 0.19.1