X2-NativeCursor-Qwen3TTS-12Hz

<div align="center"> <img src="assets/x-square-logo.png" width="140" alt="X Square Robot beaver mascot holding the moon"> <h1>X2-NativeCursor-Qwen3TTS-12Hz</h1> <p><strong>Reading-progress observer for token-level streaming Qwen3-TTS</strong></p> <p> <a href="https://github.com/X-Square-Robot/X2Streaming-TTS"><img src="https://img.shields.io/badge/GitHub-X2Streaming--TTS-black" alt="X2Streaming-TTS"></a> <a href="https://github.com/X-Square-Robot/Qwen3TTS-Streaming"><img src="https://img.shields.io/badge/Engine-Qwen3TTS--Streaming-6f42c1" alt="Qwen3TTS-Streaming"></a> <a href="https://huggingface.co/x-square-robot/X2Streaming-TTS-1.7B"><img src="https://img.shields.io/badge/Hugging%20Face-X2Streaming--TTS--1.7B-yellow" alt="X2Streaming-TTS-1.7B"></a> <a href="https://arxiv.org/abs/2609.09677"><img src="https://img.shields.io/badge/arXiv-2609.09677-b31b1b" alt="X2-NativeCursor paper"></a> </p> </div> <div align="center"> <img src="assets/native_cursor_use_cases.png" width="1000" alt="X2Streaming-TTS and X2-NativeCursor: streaming speech, continuous narration, synchronized highlighting, playback-aware interruption, and dialogue-history updates"> <p><em>Speak as text arrives, track reading progress, and use the playback clock to keep interruptions and dialogue history aligned with played audio.</em></p> </div>

Token-level streaming TTS starts speaking before the sentence is finished, so a client receives audio without knowing which characters of the source text it carries. X2-NativeCursor answers that question before waveform decoding: a 2M-parameter observer reads the codebook-0 token the Qwen3-TTS Talker emits every 80 ms, scores it against the spoken labels of the text visible so far, and publishes a cursor into the source text that never moves backward. The TTS generator, the speech tokenizer and the vocoder stay as they are; the observer is the only addition.

This repository holds the released observer head. It is integrated in the Qwen3TTS-Streaming engine (dev branch) as the native estimator behind the engine's text_progress events, and is described in the X2-NativeCursor paper and on the X2Streaming-TTS feature page.

Files

Compatibility

The head is voice- and backbone-specific. For another checkpoint, voice or codec, re-train and validate the observer before enabling native; the engine refuses a head whose SHA-256 differs from native_expected_head_sha256 when that field is set.

Usage

Download the head and place it under the engine's resources/native_cursor/:

huggingface-cli download x-square-robot/X2-NativeCursor-Qwen3TTS-12Hz \
  --local-dir ./weights/X2-NativeCursor-Qwen3TTS-12Hz
cp ./weights/X2-NativeCursor-Qwen3TTS-12Hz/qwen3_tts_12hz_la1_seed0.pt \
   <Qwen3TTS-Streaming>/resources/native_cursor/

Switch the estimator in engine.yaml, or set ENGINE_TEXT_PROGRESS_ESTIMATOR=native:

text_progress:
  estimator: native                      # ema (default) | native
  native_head_path: resources/native_cursor/qwen3_tts_12hz_la1_seed0.pt
  native_device: auto                    # auto (= cpu) | cpu | cuda | cuda:N
  native_expected_head_sha256: "aa94527fddff97cca0c337529a3b840b141d8301c4a6082557984438a7e584bc"
  native_fail_open: true                 # a load failure falls back to ema

A successful start logs:

native cursor head loaded: resources/native_cursor/qwen3_tts_12hz_la1_seed0.pt on cpu
  (labels=503, en_unit=letter, offsets=[-2,-1,0,1,2,3,4], left=29 right=1 frames,
   params=2036991, sha256=aa94527fddff)

Cursor anchors are delivered on the engine's existing text_progress events, so existing clients keep working. Each anchor carries progress_basis = native_cursor_v1 and progress_quality = aligned plus native_* diagnostic fields; the built-in ema estimator remains the fallback when the head is absent. The event contract and the design are documented upstream in docs/dev/design/native_cursor_progress.md.

How it works

1. TNPlan turns the committed text into spoken labels (the normalized reading) and records which span of the original text owns each label, so the projection back to the raw text stays exact when reading order differs from writing order (99% → 百分之九十九).
2. A native-token encoder (four dilated convolution blocks, hidden size 256) embeds each codebook-0 token with one frame of lookahead.
3. A local matcher scores the labels at offsets −2 … +4 around the previous position against the token feature and a small location state, and moves a continuous internal position by the expected offset.
4. The published cursor is the high-water mark of that position mapped to the furthest raw-text span end reached; it is monotone by construction.

Training uses onset times from Qwen3-ForcedAligner as supervision, on 20,235 utterances synthesized by the deployed custom voice, for 10 epochs (seed 0). Only the observer is trained.

Evaluation

Research evaluation (800 held-out texts in four groups of 200: plain Chinese, Chinese with numbers, Chinese with symbols, English; text supplied in 2–8 character chunks; reference = Qwen3-ForcedAligner; mean of three seeds):

Real-time factor 0.0180 against 0.3598 for the online waveform baseline.

Engine acceptance of this exact head (the engine's own audio on 80 held-out utterances, live text_progress anchors, same reference):

One native frame represents 80 ms of speech, so the observer stays inside its budget at these concurrency levels.

Limitations

• Trained for one voice and one backbone; accuracy drops on other voices (mean MAE 0.772 on three voices outside training in the research ablation). Re-train for a new voice.
• This released head uses English letter labels (en_unit=letter), as shown in Compatibility. English accuracy remains lower than Chinese in the research evaluation.
• The reference integration recomputes a 31-frame window per native frame on the frontend thread; cost grows with concurrency. Moving the observer into the engine thread with per-frame batching is the planned production path.

Citation

For X2-NativeCursor, please cite the paper:

@article{liu2026x2nativecursor,
  title   = {X2-NativeCursor: Native-Token Text Progress Tracking for Incremental-Text Streaming Codec TTS},
  author  = {Liu, Zehan and Chen, Carl and Wen, Rime and Fu, Kaiqi and Lin, Altman and Qin, Shawn and Shi, Lights and Gan, Roy and Wang, Hao and Wang, Qian},
  journal = {arXiv preprint arXiv:2609.09677},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.09677},
}

If you also use the X2Streaming-TTS streaming method, please cite:

@article{wen2026x2streamingtts,
  title   = {X2Streaming-TTS: Causal Token-Level Text-to-Speech from Streaming Text with Speech-State Inheritance},
  author  = {Wen, Rime and Liu, Zehan and Qin, Shawn and Shi, Lights and Gan, Roy and Wang, Hao and Wang, Qian},
  journal = {arXiv preprint arXiv:2608.18661},
  year    = {2026},
}

License

See LICENSE for the full Apache-2.0 terms and NOTICE for component provenance and attribution.

Apache License 2.0, Copyright (c) 2026 XSquareRobot. The head is used together with Qwen3-TTS models (Apache-2.0, Qwen team) and the Qwen3-ForcedAligner reference (Qwen3-ASR, Apache-2.0) was used only for training supervision.

中文简介

X2-NativeCursor 是一个约 2M 参数的朗读进度观察器：它读取 Qwen3-TTS Talker 每 80 ms 输出的 codebook-0 token，在波形解码之前给出"现在读到原文第几个字"，位置始终向前。 TTS 生成器、语音 tokenizer 与声码器保持原样。本仓库存放发布的观察器头，对应 x-square-robot/X2Streaming-TTS-1.7B 的音色；放进 Qwen3TTS-Streaming 引擎的 resources/native_cursor/ 并把 text_progress.estimator 设为 native 即可启用， 进度沿用现有的 text_progress 事件下发。换音色或换模型需要重新训练并验证。 许可为 Apache-2.0。

论文已公开：X2-NativeCursor: Native-Token Text Progress Tracking for Incremental-Text Streaming Codec TTS。