Downloads · 30 days
157
55% of all-time downloads
lumierenoir/DuplexCascade-PT-BR-V0
DuplexCascade-PT-BR-V0 is a machine learning model from lumierenoir. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Full-duplex speech-to-speech conversational LLM for Brazilian Portuguese (pt-BR).
Downloads · 30 days
157
55% of all-time downloads
All-time downloads
288
Public
Repo size
34.3 GB
Likes
0
Public
Click a slice to open those files.
.safetensors9.6 GB · 59%
From the Hugging Face model README
Full-duplex speech-to-speech conversational LLM for Brazilian Portuguese (pt-BR).
This is the fine-tuned conversational LLM of a DuplexCascade fork: a VAD-free cascaded ASR–LLM–TTS pipeline where a single LLM drives full-duplex spoken dialogue by emitting special tokens that control turn-taking, interruptions, backchannels and thinking pauses — without any voice-activity detection.
The model is a QLoRA fine-tune of Qwen/Qwen3-4B-Instruct-2507 on synthetic pt-BR duplex micro-turn data, trained with the weighted/masked cross-entropy objective from the DuplexCascade paper (§4.1). This repo also hosts DuplexCascade-Distill, a second model that learns the same protocol without the capability regression of plain SFT — see DuplexCascade-Distill below.
| Path | Description |
|---|---|
models/adapter/ | QLoRA adapter of the SFT baseline, V0 (LoRA weights + tokenizer + training_args.bin) — resume fine-tuning from here |
models/export/gguf/DuplexCascade-PT-q4_k_m.gguf | SFT baseline GGUF q4_k_m quantization for llama.cpp serving |
data/dialogues_combined.jsonl | Synthetic pt-BR dialogue corpus (short + long replies) used for training |
models/distill/adapter/ | QLoRA adapter of DuplexCascade-Distill (resumable fine-tune state) |
models/distill/export/gguf/DuplexCascade-Distill-q4_k_m.gguf | Distill model GGUF q4_k_m for llama.cpp serving |
models/distill/export/gguf/Modelfile | Ollama Modelfile: serving recipe + why the distill model exists and its results |
data/distill/teacher_duplex_train.jsonl | Distill fine-tuning data: 10k pt-BR dialogues, assistant content regenerated by the frozen base under the duplex trigger, converted to duplex micro-turns |
data/distill/anchor_prompts.jsonl | Capability-anchor prompts (function calling / general QA / reasoning) for the distill KL term |
Builders and training code live in the source repos (duplex_cascade, duplex_cascade_distill).
The model is trained to consume interleaved micro-turns supervised with special tokens:
| Token | Meaning |
|---|---|
| < | user is speaking |
| `< | user finish speaking |
| `< | no voice |
| `< | user is thinking |
| `< | user interruption |
| `< | user backchannel |
| `< | system backchannel |
Loss is computed only on system micro-turns, and each special token carries a task-specific loss weight (paper §4.1): <|user finish speaking|> ×10, <|user interruption|> ×5, <|system backchannel|> ×3, <|user backchannel|> ×2, <|user is speaking|>/<|user is thinking|> ×1.
At inference the LLM emits these tokens to negotiate the conversation flow in real time — no VAD, no fixed turn-taking.
Qwen/Qwen3-4B-Instruct-2507 (bf16, dense)r=16, α=16, dropout 0, on all linear modules (q/k/v/o/gate/up/down), plus modules_to_save=["embed_tokens", "lm_head"] so the new special-token embeddings are trainedDuplexSFTTrainer implementing weighted/masked cross-entropy over system micro-turns onlyllama-server -m DuplexCascade-PT-q4_k_m.gguf \
--port 8080 -sp \
--jinja --chat-template chat_template.jinja
-sp(special-token emission) is required — without it the duplex special tokens are silently dropped from completions.
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")
model = PeftModel.from_pretrained(base, "lumierenoir/DuplexCascade-PT-BR-V0/models/adapter")
tokenizer = AutoTokenizer.from_pretrained("lumierenoir/DuplexCascade-PT-BR-V0/models/adapter")
The full stack (llama.cpp + faster-whisper STT + pocket-tts TTS + web UI) is orchestrated by run_services.sh in the source project.
models/distill/… + data/distill/… hold the second model in this repo.
Plain SFT on the duplex protocol regresses general capabilities: the V0 baseline above learns turn-taking (98.0% tag accuracy) but its function calling collapses to 55.6% and general pt-BR instruction-following to 41.7% — textbook catastrophic forgetting under off-policy SFT.
DuplexCascade-Distill replaces the hard content fit with a self-distillation objective (SDFT, arXiv:2601.19897, adapted without RL or on-policy rollouts): the assistant content is generated by the frozen base model itself prompted with a duplex trigger; during fine-tuning the 7 special tags are learned by weighted CE while every other token position is pulled toward the frozen base's own distribution via per-token forward-KL (disable_adapters() teacher pass, offline top-k=32 cache). The KL term pins "everything else" to the pretrained model — that is the capability-retention mechanism.
| Model | Function calling (18 pt-BR tasks) | General (12 pt-BR tasks) | Tag accuracy | KL drift |
|---|---|---|---|---|
| base | 100% | 100% | 0.0% | 0.51* |
| distill (this repo) | 100% | 100% | 96.4% | 1.07 |
| sft (V0 baseline) | 55.6% | 41.7% | 98.0% | 4.76 |
* top-k truncation floor of a perfectly-matched model (reference).
Standard benchmarks (2000-example subsets): distill stays within ~1–2 pp of base on HellaSwag (0.566 vs 0.584), ARC-Challenge (0.563 vs 0.584) and MMLU (0.698 vs 0.705); the SFT baseline loses 4.5 pp on MMLU (0.660). Extra benchmarks: distill even beats base on TruthfulQA mc2 (0.649 vs 0.626) and IFEval loose (0.762 vs 0.730); its single honest regression is GSM8K long-form CoT (0.777 vs 0.890). The SFT baseline drops on all of them.
Full tables and reproduction commands: eval/RESULTS.md.
llama-server -m DuplexCascade-Distill-q4_k_m.gguf \
--port 8080 -sp \
--repeat-penalty 1.15
-sp is required so duplex special tokens appear in completions.models/distill/adapter/ onto Qwen/Qwen3-4B-Instruct-2507 with PEFT and continue with training/train_distill.py.The distill model occasionally opens its reply with the continuation of the user's own sentence instead of a fresh answer — e.g. user: "Qual é a capital do Brasil?" → assistant: ", a capital do Brasil é Brasília." (leading comma
Why: the teacher (frozen base) sometimes autocompleted the user turn when generating the assistant content under the v1 "Responda de forma curta" trigger, and the content forward-KL anchors that mode into the trained model. The v2 trigger at inference reduces it but does not remove it.
Current mitigation: the bridge strips leading whitespace + continuation
punctuation from every micro-turn before TTS/transcript/history
(_strip_leading_continuation in distill/DuplexCascade/server.py).
Planned fix (future): regenerate the teacher training data with the v2
trigger ("responda de forma natural e completa … que responda de fato ao que o
usuário perguntou … não se repita") and retrain — see distill/SPEC.md §6 R0 —
so the continuation artifact is gone from the model natively instead of being
stripped at inference.
Based on:
DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR–LLM–TTS Pipeline and Micro-Turn Optimization — Jianing Yang, Yusuke Fujita, Yui Sudo (sbintuitions).
The third model in this repo: a 0.8B protocol companion that attaches MTP-style full-duplex turn-taking to any conversational LLM — without fine-tuning the big model. It plays the role of a controller that predicts, step by step, only the control tags of the duplex protocol at the current point of the conversation; the big model just writes the answer text.
| Tag | Meaning |
|---|---|
| `< | user is speaking |
| `< | user finish speaking |
| `< | user interruption |
| `< | user backchannel |
| `< | user is thinking |
| `< | system take floor |
| `< | system handover |
The last two tags implement the promptable controller: the companion
reads a system instruction (the rule, --policy-rule) and fires when the
rule's trigger appears in the user's words — e.g. the CPF rule: "No CPF,
apenas números são permitidos. Se o cliente disser uma letra em vez de um
número, interrompa..." — the letter is detected mid-dictation and the system
cuts the user off instantly. The model learns the number-vs-letter concept
from training on the full alphabet and a rich number-word set; probes confirm
unseen letters fire and that the system tags never fire without a rule.
Trained with the SDFT distill objective (tag CE at boundaries + forward-KL to the frozen base at content positions, λ=0.3, top-k 32) so the language and instruction-following capability survive the protocol fine-tuning.
| Path | Description |
|---|---|
models/mtp_like/checkpoint_v6/ | Final training checkpoint (step 3000): LoRA adapter + tokenizer + policy_cfg.json (tag vocabulary + base model) |
models/mtp_like/export/gguf/MTP-Like-Companion-v6-Q8_0.gguf | Merged bf16 companion quantized to Q8_0 (812 MB) for llama.cpp/vLLM serving |
data/mtp_like_snapshot/ | Snapshot of the training data (15k stratified train records across all 5 record kinds + the full 4.7k-record eval split used for the numbers below) — the full dataset is 22 GB; this snapshot reproduces the data distribution |
| tag | P | R | F1 |
|---|---|---|---|
| no tag | 1.00 | 0.95 | 0.973 |
| user is speaking | 0.99 | 0.92 | 0.953 |
| user finish speaking | 0.79 | 0.98 | 0.877 |
| user interruption | 0.95 | 0.99 | 0.969 |
| user backchannel | 1.00 | 0.98 | 0.99 |
| user is thinking | 0.90 | 1.00 | 0.949 |
| system take floor | 1.00 | 1.00 | 1.00 |
| system handover | 1.00 | 1.00 | 1.00 |
Overall tag accuracy 0.956. Controller probes: barge-in → interruption (0.99); backchannel → not a barge-in; CPF rule + letter (dictation rhythm) → take_floor (1.00); no rule + letter → never fires.
Run the live browser demo (STT + TTS + bridge) from the source repo:
./run_services.sh # opens http://localhost:31606 — speak in pt-BR
Say "quero um cartão de crédito" → the big model asks for the CPF (it keeps
conversation history); dictate "1, 2, 3, x, 5" → the companion fires
<|system take floor|> on the letter and the system interrupts you with
"Apenas números são aceitos no CPF.".
The GGUF serves the same companion under llama.cpp (quantized Q8_0) — the 0.8B model is small enough for real-time control on a laptop.
Code, training recipes and the decision log: mtp_like/.