Kiseki-TTS

Kiseki-TTS

A small, fast Japanese TTS model with a Mamba2 state-space decoder, built on top of Qwen/Qwen3-TTS-Tokenizer-12Hz.

Kiseki-TTS generates discrete neural audio codec tokens at 12.5 Hz and decodes them to waveform with the Qwen3 TTS codec. Because the acoustic decoder is a linear-time SSM rather than a self-attention stack, generation cost is constant per frame — memory does not grow with utterance length, and there is no KV cache to manage.

The same checkpoint also performs ASR (Japanese speech → text), since TTS and ASR were trained jointly in a single multi-task run.

Example

<audio controls> <source src="https://huggingface.co/telecomadm1145/Kiseki-TTS/resolve/main/example_output.wav" type="audio/wav"> </audio>

---

Model Details

Model Description

• Developed by: telecomadm1145
• Model type: Encoder–decoder (Transformer encoder + cross-attention/Mamba2 decoder)
• Language: Japanese (ja)
• License: MIT
• Finetuned from: telecomadm1145/Kiseki-1.1-0.3B (a seq2seq translation model)
• Audio codec: Qwen/Qwen3-TTS-Tokenizer-12Hz

Architecture

The decoder deliberately has no causal self-attention. Temporal context is carried entirely by the Mamba2 recurrent state; text conditioning enters through cross-attention whose K/V are computed once from the encoder and reused for every frame.

Audio tokenization

Depth modelling (MTP head). Each frame's 16 codebook layers are predicted by a shared "multi-token prediction" head rather than 16 separate decoder passes. Layer q sees the decoder hidden state plus the exclusive prefix sum of the embeddings of layers 0 … q-1:

logits_q = W_q · Block( h_t + (1/√Q) · Σ_{j<q} E_j(c_t^j) ) + b_q

where Block is a small RMSNorm → SwiGLU(×2) → RMSNorm residual body shared across all 16 layers. This means one trunk evaluation per frame and 16 cheap head evaluations, instead of 16 full autoregressive steps.

---

Why it's fast

1. 12.5 Hz is the headline number. One second of speech is 12.5 decoder steps. Codecs running at 50 Hz or 75 Hz need 4–6× more autoregressive steps for the same audio. Concretely:

A 10-second utterance is 125 recurrent steps. For comparison, a token-level LLM TTS at 50 Hz × 8 codebooks would be pushing ~500 trunk steps for the same clip.

2. O(1) state, not O(T) cache. The Mamba2 decoder carries a fixed (32 heads × 128 state × 64 dim) tensor plus a 3-frame conv window per layer. Generating 40 seconds costs exactly as much per step as generating 1 second — no attention matrix, no KV cache reallocation, no quadratic blowup. Long-form synthesis degrades gracefully instead of falling off a memory cliff.

3. Cross-attention K/V is computed once. Encoder output is projected to per-layer K/V a single time during prefill. Every subsequent frame does one small Q·Kᵀ against a fixed-length text sequence.

4. A shallow decoder. Only 6 decoder layers sit in the autoregressive loop. The 12-layer encoder runs exactly once, fully parallel over the input text.

5. The depth loop is cheap. The 16 codebook layers are resolved sequentially (layer q conditions on layers <q), but each step is one ×2 SwiGLU block at d=1024 — small enough that batch-1 generation is memory-bandwidth-bound rather than compute-bound.

---

How to Get Started

Setup

import torch
from transformers import AutoModelForSeq2SeqLM, PreTrainedTokenizerFast

repo = "telecomadm1145/Kiseki-TTS"
tok = PreTrainedTokenizerFast.from_pretrained(repo)
m = AutoModelForSeq2SeqLM.from_pretrained(repo, trust_remote_code=True).eval().cuda()

lang = tok.convert_tokens_to_ids("<|2ja|>")

TTS — text to audio codes

ids = m.build_tts_input_ids(tok.encode("こんにちは"), lang, device="cuda")
out = m.generate_audio(ids, max_new_frames=250, temperature=0.9, top_k=50)
codes = out["audio_codes"][0][out["valid_mask"][0]]   # (T, Q)

build_tts_input_ids assembles [TTS] <|2ja|> …text… <eos>. max_new_frames=250 ≈ 20 seconds of audio at 12.5 Hz.

Decode codes to waveform

import soundfile as sf
from qwen_tts import Qwen3TTSTokenizer

tokenizer = Qwen3TTSTokenizer.from_pretrained(
    "Qwen/Qwen3-TTS-Tokenizer-12Hz",
    device_map="cuda:0",
)

wavs, sr = tokenizer.decode({"audio_codes": codes})
sf.write("decode_output.wav", wavs[0], sr)

ASR — audio codes to text

ASR consumes only the first quantizer layer (q0) of the codec output.

enc = m.build_asr_encoder_codes(q0_codes, device="cuda")
txt_ids = m.generate_transcription(enc, decoder_start_tokens=[m.config.bos_token_id, lang])
print(tok.decode(txt_ids[0], skip_special_tokens=True))

Sampling parameters

Generation stops when layer 0 emits EOS (2048). BOS and PAD are masked out of the logits, so they can never be sampled.

---

Training Details

• Initialization: all non-audio weights restored from the Kiseki-1.1-0.3B translation checkpoint; the audio embedding tables and MTP head were randomly initialized and trained with a 3× learning-rate multiplier relative to the backbone.
• Objective: joint TTS + ASR, sampled at roughly 70 % TTS / 30 % ASR per step.
• TTS supervision: teacher forcing in both directions — along time (previous frames) and along depth (ground-truth prefix of lower codebook layers).
• Packing: multiple utterances are packed per row with segment-ID masking; attention, the depthwise conv, and the SSM recurrence are all reset at segment boundaries so packed samples never leak into one another.
• Optimizer: AdamW, cosine schedule with warmup, gradient clipping at 1.0, weight decay applied only to ≥2-D parameters.
• Data: telecomadm1145/asmr_archive_qwentts_encoded.
• Task tokens: [TTS] and [ASR] reuse the last two UL2 sentinel IDs in the 65,792-token vocabulary, so the tokenizer is unchanged from the base model.

---

Limitations and Bias

• Japanese only. No other language was trained; the <|2ja|> tag is the only supported language token for speech tasks.
• Single-domain voice. Training data is ASMR-style Japanese speech, so timbre, pacing, and recording character are strongly biased toward that domain. There is no speaker conditioning or voice cloning — output voice is not controllable.
• ~41 s ceiling. The model saw at most 512 frames during training. Longer requests will run (the SSM state is length-agnostic) but quality beyond ~40 s is untested.
• ASR is a byproduct. It reads only quantizer layer 0 and was trained as an auxiliary task; do not expect dedicated-ASR accuracy.
• Sampling sensitivity. Discrete codec TTS can occasionally loop or emit early EOS. Lower temperature_q0 if you observe repetition.
• No safety filtering was applied to the training corpus.

---

Special Token Reference

---

Citation

@misc{kiseki-tts,
  title  = {Kiseki-TTS: A Fast Japanese TTS Model with a Mamba2 Decoder},
  author = {telecomadm1145},
  year   = {2026},
  url    = {https://huggingface.co/telecomadm1145/Kiseki-TTS}
}