vits-csmsc

Model Card for VITS-CSMSC (IB-Robot, hybrid CPU + Ascend 310P)

PaddleSpeech VITS-CSMSC — single-speaker Mandarin text-to-speech, 22.05 kHz — packaged for the IB-Robot framework as a hybrid deployment: the text encoder and duration predictor run as ONNX on CPU, and the flow decoder (which contains the vocoder and dominates compute) runs as an .om on Ascend 310P1.

This is the speech-output stage of the robot pipeline. Accepts arbitrary Chinese text; the voice is the fixed CSMSC speaker.

Deployments

Two fixed-shape flow decoders also ship in the repo without their own deployment entry, matching the convention across this organisation that one hardware target carries one deployment:

Both are declared in bundle.files and load by pointing an ACL session at the file directly. They accept exactly their frame count, so an utterance has to be padded or trimmed to fit; the dynamic build takes any length, which is why it is the one wired to ascend_310p. Accuracy for all three is in Validation — they are within 1e-5 cosine of each other.

The ascend_310p row is a composite deployment in the manifest: there is no single runtime_profile, because there is no single backend. role_identities and role_runtime_profiles name the backend per role — onnx on CPU for text_encoder and duration, ascend/acl for flow_decoder. torch_cpu runs all three roles on ONNX Runtime and so keeps one shared profile.

The two host-side steps between duration and flow (duration_expand_path, sample_z_p) are listed in each deployment's metadata.host_stages; they are not model roles and have no artifact.

Shared contract: tensor_model/vits/synthesize (tts.phone_ids [1,-1] -> tts.audio [1,-1]), preprocessing paddlespeech-zh-frontend-jieba-pypinyin-phoneid-v1, output mono-float32-pcm-22050hz-v1, logical revision paddlespeech-vits-csmsc@onnx-export-v1.

ascend_310p uses the dynamic-shape flow decoder, so no padding or trimming is needed. For latency-sensitive use with a known utterance length, load one of the two fixed-shape builds above directly.

Pipeline

Chinese text
  → PaddleSpeech frontend / pypinyin      → phone ids
  → text encoder     (ONNX, CPU)          → x, m_p, logs_p (192-d), x_mask
  → duration predictor (ONNX, CPU)        → logw
  → host: alignment expansion + z_p       (see below)
  → flow decoder     (.om, Ascend 310P1)  → waveform
  → 22.05 kHz mono wav

Host stage (between duration and flow)

This is not a thin glue layer — it is part of the model's semantics and must be reproduced exactly. See assets/reference_impl/run_vits_hybrid_310p.py:

dur    = ceil(exp(logw) * float32(x_mask))  # per-phone frame counts
path   = generate_path(dur, ...)           # [1, T_text, T_frames] alignment
m      = (path.transpose @ m_p.transpose).transpose      # expand to frames
logs   = (path.transpose @ logs_p.transpose).transpose
y_mask = ones([1, 1, T_frames])
z_p    = m + randn_like(m) * exp(logs) * 0.667           # noise_scale 0.667

The reference uses seed 1234 for reproducibility; the sampling of z_p is genuinely stochastic, so runs with different seeds produce different (equally valid) audio, and bit-exact comparison against a reference requires pinning it.

Tensor contract

text encoder   text        int64  [1, -1]      phone ids
               text_lengths int64  [1]
             → x           float32 [1, 192, -1]   outputs are positional and
               m_p         float32 [1, 192, -1]   come back in this order
               logs_p      float32 [1, 192, -1]
               x_mask      bool    [1, 1, -1]

duration       x           float32 [1, 192, -1]
               x_mask      bool    [1, 1, -1]     bool, not float
             → logw        float32 [1, 1, -1]

flow decoder   z_p         float32 [1, 192, T]   T = 639 / 681 / -1
               y_mask      float32 [1, 1, T]
             → wav         float32 [1, T*256]    256 samples per frame

Sample rate 22050 Hz, hop 256 → one frame ≈ 11.61 ms.

Repository Structure

• inference_manifest.json — deployment routing (schema v3)
• assets/adapter.json — adapter identity (vits / synthesize)
• assets/phone_id_map.txt — 268-entry phone→id map used by the frontend
• assets/snapshot_iter_150000.pdz — the upstream Paddle training checkpoint (1.1 GB), byte-identical to the one in vits_csmsc_ckpt_1.4.0.zip
• assets/default.yaml — that checkpoint's own training config
• assets/reference_impl/run_vits_hybrid_310p.py — the hybrid runner, and the normative source for the host stage above
• assets/reference_impl/synthesize_vits_310p.py — text → wav CLI
• artifacts/onnx/*.onnx — the three ONNX graphs (CPU stages + CPU reference)
• artifacts/ascend_310p/*.om — three flow decoder builds

Source Model

Upstream is PaddleSpeech VITS-CSMSC, released checkpoint vits_csmsc_ckpt_1.4.0. The manifest's logical revision is paddlespeech-vits-csmsc@onnx-export-v1 — it names the ONNX export this bundle runs, not the upstream release tag, which is 1.4.0.

wget https://paddlespeech.cdn.bcebos.com/Parakeet/released_models/vits/\
vits_csmsc_ckpt_1.4.0.zip
# 1,027,863,302 bytes
# sha256 994340036777b24f3bf02967bf37f570d40e5514b54ff0001c5b4732921ae064

The archive holds exactly three files, and all three are shipped here byte-for-byte (sha256 verified against a fresh download and extraction):

phone_id_map.txt pins the phone inventory, and therefore the text frontend contract, to this exact release. default.yaml is the checkpoint's own training config and independently confirms the numbers this bundle's tensor contract depends on: fs: 22050, n_shift: 256 (the 256 samples per frame above) and hidden_channels: 192 (the 192-d z_p).

Shipping the checkpoint follows the convention in the rest of this organisation, where the upstream weights live under assets/. Note that it is not the artifact any deployment loads — every deployment runs the ONNX graphs below. It is there so the bundle carries its own provenance.

The three ONNX graphs were exported from that checkpoint in separate passes, which is why their opsets differ (12 / 13 / 11):

Shipping the checkpoint does not make the ONNX byte-reproducible from it: the export script is not part of this bundle, and PaddleSpeech static-model export behaviour changed with PaddlePaddle ≥ 2.5. What is established is the accuracy of the export — cosine 0.9999999999998159 against the Paddle reference, below — plus the exact checkpoint and config it came from.

Validation

All measured on a real Ascend 310P1, against ONNX Runtime CPU fp32.

ONNX export vs Paddle reference (text encoder + duration) — the export itself is lossless:

cosine   0.9999999999998159
max_abs  4.291534423828125e-06

Flow decoder OM vs ONNX, on the flow subgraph alone:

Those are the conversion-time numbers. They have since been independently reproduced on the shipped artifacts — same board, same artifacts/onnx/ vits_flow_decoder.onnx reference, but with z_p built through the real encoder → duration → host-alignment path rather than synthetic tensors (_tools/board/verify_vits_flow_om_vs_onnx.py):

Latency matches the conversion report to within 0.3 ms on every build, and accuracy comes out slightly better on realistic z_p than on the synthetic input used at conversion time. The dynamic build's higher RTF at 399 frames is the fixed per-call overhead spread over less audio, not a slower kernel — at 639 frames it lands within ~40 ms of the fixed build (see below).

End-to-end hybrid pipeline, 20 runs, 7.42 s of output audio. Measured with the fixed t639 build, which is what the reference runner loads (assets/reference_impl/run_vits_hybrid_310p.py:15); the same pipeline on the ascend_310p dynamic build is the baseline row two tables down, 367.63 ms:

cosine_vs_onnx   0.9999853236698978
max_abs          0.008778512477874756
latency  mean    327.38 ms   p50 323.81 ms   p95 355.89 ms
RTF              0.0441

Per-stage breakdown (mean, ms):

The host alignment stage is now the single largest cost — larger than the NPU work it feeds. It is pure numpy (generate_path plus two matmuls) and is the obvious next optimisation target; moving it onto the NPU or vectorising it would cut end-to-end latency by roughly a third.

Dynamic build, variable text length:

Latency grows sub-linearly with length, so RTF improves on longer utterances. The dynamic build costs ~40 ms over the fixed t639 build at the same length.

Usage

from inference_manifest import load_inference_manifest

validated = load_inference_manifest("models/vits-csmsc", "ascend_310p")

The hybrid path needs both an ONNX Runtime CPU session (encoder, duration) and an ACL session (flow), plus the host stage between them — see assets/reference_impl/run_vits_hybrid_310p.py, which is the normative implementation.

Known Limitations

• Not a pure-NPU TTS. Text encoder and duration remain on CPU; only the flow decoder is compiled to OM.
• Chinese only. English and other languages are not a validation target.
• Fixed speaker. Single-speaker CSMSC voice, no speaker conditioning.
• The two extra fixed-shape OMs require exact frame counts (639 / 681). The ascend_310p deployment uses the dynamic build and has no such limit; the fixed builds are for callers who know their length and want the ~40 ms.
• Output is stochastic (noise_scale = 0.667) unless the RNG seed is pinned.

Environment

Validated on: Ascend 310P1 with CANN and ais_bench, Python 3.11.6, ONNX Runtime 1.23.2.

License

Code and packaging: Apache-2.0. Model weights follow the upstream PaddleSpeech license (Apache-2.0); the CSMSC corpus is subject to its own terms.

Citation

@inproceedings{vits2021,
  title     = {Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech},
  author    = {Kim, Jaehyeon and Kong, Jungil and Son, Juhee},
  booktitle = {arXiv preprint arXiv:2106.06103},
  year      = {2021}
}
@article{paddlespeech2022,
  title   = {PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit},
  author  = {Zhang, Hui and Yuan, Tian and Chen, Junkun and Li, Xintong and Zheng, Renjie and Huang, Yuxin and Chen, Xiaojie and Gong, Enlei and Chen, Zeyu and Hu, Xiaoguang and Yu, Dianhai and Ma, Yanjun and Huang, Liang},
  journal = {arXiv preprint arXiv:2205.12007},
  year    = {2022}
}
@software{ib_robot,
  title   = {IB-Robot: Intelligence Boom Robot},
  url     = {https://atomgit.com/openeuler/IB_Robot},
  license = {Apache-2.0}
}