MiniCPM-o-4_5-tts-dflash

MiniCPM-o 4.5 TTS DFlash Drafter

This repository hosts the DFlash draft checkpoint that accelerates the Talker/TTS decoder of MiniCPM-o 4.5 through speculative decoding. The drafter is not a standalone speech model. In each round it proposes a block of audio codec tokens from the Talker's internal features, the 20-layer Talker verifies the whole block in a single forward pass, and the longest correct prefix is accepted. Under the default configuration the acceptance rule is distribution-preserving, so the generated speech follows exactly the same distribution as the unaccelerated Talker while the codec-token decoding runs faster.

The inference code that loads, validates and runs this checkpoint is maintained separately.

Code repository

The corresponding code lives at https://github.com/cs-qyzhang/minicpm-o-4.5-tts-dflash.

This checkpoint is the default --draft-model consumed by that repository. It provides three command-line entries that share the same drafter:

• minicpmo-dflash-simplex: text, image or video input is turned into speech through the official model.chat() path;
• minicpmo-dflash-duplex: video and audio are streamed in one-second chunks while the model keeps listening and speaking, producing a time-aligned WAV, MP4 and JSON log;
• minicpmo-dflash-benchmark: the DFlash acceptance rate and accepted length per round are measured over a built-in suite of simplex samples.

The loading logic is a self-contained PyTorch implementation of the Qwen3-style DFlash layers (src/minicpmo_dflash/draft.py); it reads config.json and model.safetensors from this directory directly and does not require speculators to be installed at runtime. Training details, corpus construction and ablations are recorded in training.md of the code repository, and the full training run is logged publicly in the wandb project.

Base model and scope

The verifier is the Talker of MiniCPM-o 4.5: an autoregressive Llama-architecture decoder with 20 layers and hidden size 768, whose vocabulary holds 6562 audio codec tokens (the last id, 6561, is the codec EOS). DFlash accelerates only the TTS codec-token decoding performed by the Talker. Text decoding by the main Qwen3 Thinker, as well as the vision and audio encoders, still runs the original Hugging Face Transformers remote code and is unaffected by this checkpoint.

Architecture

The drafter is a one-pass anchored-block DFlash model built on a Qwen3 decoder. Each speculative round feeds a block of block_size slots arranged as [anchor, <MASK> × (block_size-1)], with bidirectional attention inside the block and visibility of the context preceding the anchor. The verifier's hidden states from five auxiliary layers are compressed by fc + RMSNorm and injected into the K/V of every draft layer, which is how the drafter perceives the verifier's internal features. One forward pass fills in the seven speculative tokens of the block. The anchor slot receives the Talker's per-position input embedding rather than a token id, since that embedding is not a plain lookup: it combines the text embedding with the projected Thinker hidden state. This is the main difference between the training of this drafter and conventional LLM drafter training.

The configuration recorded in config.json is summarized below.

The checkpoint holds about 43.7M tensor elements in total. The embed_tokens (6563 × 768) and lm_head (6562 × 768) are copied from the Talker and frozen; the trainable drafter — fc, the two norms and the four decoder layers — is roughly 30M parameters. The d2t / t2d buffers store the draft-to-target and target-to-draft vocabulary mappings. In this checkpoint they are identity mappings, so draft ids 0..6561 correspond one-to-one with codec ids 0..6561 and the mask token 6562 is not part of the prediction vocabulary.

Files

The code repository's own loader reads config.json and model.safetensors directly, so those two files are the only ones required at inference time; it validates that speculators_model_type is dflash, that sample_from_anchor is false, and that the t2d / d2t mappings are consistent.

Usage

This checkpoint is loaded by the code repository, not by a Hugging Face AutoModel pipeline. Install the environment first (an NVIDIA GPU and a system FFmpeg are required):

git clone https://github.com/cs-qyzhang/minicpm-o-4.5-tts-dflash
cd minicpm-o-4.5-tts-dflash
uv sync

--draft-model accepts either the Hugging Face repo id cs-qyzhang/MiniCPM-o-4_5-tts-dflash or a local directory; when a repo id is given, the first run downloads it into the HF cache.

# simplex: text -> speech with distribution-preserving speculative decoding
uv run minicpmo-dflash-simplex \
  --draft-model cs-qyzhang/MiniCPM-o-4_5-tts-dflash \
  --text "你好，欢迎使用 MiniCPM-o 四点五 DFlash 语音合成。" \
  --tts-draft-temperature 0.8 \
  --output outputs/simplex_dflash.wav

# duplex: streaming video + audio
uv run minicpmo-dflash-duplex \
  --draft-model cs-qyzhang/MiniCPM-o-4_5-tts-dflash \
  --video ./omni_duplex1.mp4

# benchmark: per-sample acceptance rate / EAL / tokens-per-round
uv run minicpmo-dflash-benchmark \
  --draft-model cs-qyzhang/MiniCPM-o-4_5-tts-dflash

The draft proposal temperature is the only non-official sampling parameter introduced by this work. It defaults to 0.8, may be set to -1 to follow the verifier temperature, or to 0 to propose with argmax. Because acceptance is distribution-preserving by default, this temperature affects only the acceptance rate and the speed, never the output distribution. --disable-dflash skips this checkpoint entirely and falls back to the model's native Transformers TTS decoder.

Benchmark

Metrics measured end to end with minicpmo-dflash-benchmark over the 20 built-in simplex samples (seed 42), draft temperature 0.8, verifier temperature 0.8 over the full codec vocabulary, distribution-preserving acceptance (accept-top-k=0, accept-min-prob=0) and seven speculative tokens per round. EAL is the number of draft tokens accepted per round on average, out of the seven proposed; the acceptance rate, EAL and tokens/round are micro-averaged over rounds.

Training

The drafter was trained with a modified speculators DFlash pipeline on about 459k teacher-forcing samples recorded from the MiniCPM-o 4.5 Talker over five Chinese and English utterance corpora (DailyDialog, Topical-Chat, NaturalConv, LCCC and a single-line subset of open-perfectblend). Training used a KL loss between the draft and teacher logits with fixed exponential positional decay, the Muon optimizer at learning rate 3e-4, and eight epochs, keeping the lowest-validation-loss checkpoint. The corpus mixture, the number of auxiliary layers and the decay coefficient were not tuned exhaustively, and neither multimodal input nor duplex mode took part in training, although the drafter performs well under both in practice. The full recipe is documented in training.md.

License and attribution

The drafter weights include embed_tokens and lm_head copied from the MiniCPM-o 4.5 Talker, so the use of this checkpoint is subject to the license and terms of the base model openbmb/MiniCPM-o-4_5. Please refer to the base model repository for its licensing conditions and cite it accordingly when using this drafter.