sopho-poetry-tts

sopho-poetry-tts (m3_v6)

On-device Chinese classical-poetry TTS for the SuFei Android app. A compact FastSpeech2 acoustic model (7.6 M params) plus the PaddleSpeech HiFi-GAN CSMSC vocoder, both exported to ONNX. Total deploy size ~68 MB, runs on CPU (onnxruntime) with no network at runtime.

Files

SHA256SUMS.txt carries the canonical checksums; the app's Gradle fetchTtsModels task pins this revision and verifies each file's sha256 before bundling it.

Pipeline

text → G2P (PaddleSpeech phones) → phone_ids
     → fastspeech2_sufei.onnx → mel[T,80]
     → * mel_std + mel_mean  (norm_stats.npz)
     → hifigan_csmsc.onnx → 24 kHz pcm

• Sample rate 24000 Hz, mel bins 80, phone vocab 268.
• G2P convention: PaddleSpeech phones; the interpunct · produces no token (the m3_v6 dot-token fix); punctuation ，。？！ kept, ；： → ，.

On-device notes (fp32 + mobile-safe length regulator)

Two changes make this model run correctly under onnxruntime-android:

1. Length regulator — exported with a vectorized cumsum + GreaterOrEqual-sum + Gather formulation instead of torch.repeat_interleave. The latter exports to Loop + SplitToSequence / ConcatFromSequence; the vectorized form uses only standard tensor ops and is numerically identical (max|diff| = 0).
2. fp32, not fp16 — an fp16 FS2 yields a 0-frame mel on the ARM onnxruntime-android CPU EP (the fp16 duration path collapses), while the same fp16 graph is correct on x86. fp32 is exact and verified on-device (instrumented NarOnnxEngineTest). fp16 halves the acoustic model but is unusable on the target runtime, so fp32 ships.

Quality

ASR-CER (FunASR), holdout-20 / train-30 seed42: train 20.6 % / holdout 21.0 % (fp32 baseline; production v9 was 24.0 % / 21.5 %). Checkpoint m3_v6_final.pt (step 30000).

Training data

Distilled from the sopho-poetry-tts-data cosyvoice3/ lineage (Fun-CosyVoice3-0.5B teacher).

License

Apache-2.0. The vocoder hifigan_csmsc.onnx is PaddleSpeech's pretrained HiFi-GAN CSMSC, redistributed unchanged under Apache-2.0; the FastSpeech2 acoustic model was trained by this project. Teacher audio was produced by Apache-2.0 models, which place no restriction on their outputs.