Downloads · 30 days
0
harvestsu/qwen3-tts-0.6b-customvoice-jetson-trtllm-int4fp8
qwen3-tts-0.6b-customvoice-jetson-trtllm-int4fp8 is a machine learning model from harvestsu. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Production deploy bundle for the CustomVoice Qwen3-TTS-12Hz-0.6B talker, quantized for NVIDIA Jetson Orin (Orin NX, JetPack 6 / CUDA 12.6, TensorRT-Edge-LLM v0.8.0).
Downloads · 30 days
0
Access
Public
Updated Jun 21, 2026
Repo size
936 MB
Likes
0
Public
Click a slice to open those files.
.tgz936 MB · 100%
From the Hugging Face model README
Production deploy bundle for the CustomVoice Qwen3-TTS-12Hz-0.6B talker, quantized for NVIDIA Jetson Orin (Orin NX, JetPack 6 / CUDA 12.6, TensorRT-Edge-LLM v0.8.0).
This is the int4-AWQ talker + fp8 text-embedding precision configuration — the smallest intelligible CustomVoice TTS engine set, ~960 MB smaller per instance than the all-fp16 build. ASR-verified intelligible (see Validation).
Parallel bundles:
harvestsu/qwen3-tts-0.6b-base-jetson-trtllm-int4fp8 (the base speaker stack)harvestsu/qwen3-asr-0.6b-int4-v080 (the ASR stack)The worker MUST be pointed at the CustomVoice talker tokenizer shipped in this bundle
(models/qwen3-tts-customvoice/tokenizer/). Its processed_chat_template.json has a
non-empty assistant role:
"assistant": { "prefix": "<|im_start|>assistant\n", "suffix": "<|im_end|>\n" }
If you instead point the worker at a Qwen3-ASR tokenizer (or any template whose assistant
role prefix/suffix is empty ""), the assistant turn contributes zero prompt tokens. The
talker then gets a degenerate prefill (prompt length N <= 0) → garbage codec tokens →
garbled / silent audio.
The rebuilt worker in this bundle (md5 a9d1e6da) adds a hard N <= 0 prefill guard that
aborts loudly instead of emitting garbage — but the right fix is always to pass the correct
tokenizer dir. Verify before deploy:
python3 -c "import json;a=json.load(open('models/qwen3-tts-customvoice/tokenizer/processed_chat_template.json'))['roles']['assistant'];assert a['prefix'] and a['suffix'];print('OK',a)"
| Component | Precision | Size |
|---|---|---|
| Talker LLM engine | int4-AWQ | 246 MB |
Talker text_embedding | fp8 (E4M3, per-group-128 amax/448) | 321 MB |
Talker embedding / text_projection (sidecars) | fp16 | 6.3 MB / 12.6 MB |
Code predictor (cp_dir) | fp16 | ~182 MB |
| Code2wav vocoder | fp16 | ~224 MB |
vs an all-fp16 talker (fp16 LLM engine ~745 MB + fp16 text_embedding ~610 MB), this config saves ≈ 960 MB per running instance — directly multiplying available headroom for N>1 concurrency on 8 GB Orin.
cv-customvoice-int4fp8-deploy/
├── README.md
├── jv-workers/
│ ├── qwen3_tts_worker # rebuilt worker (N<=0 guard + fp8-embed wiring)
│ └── libNvInfer_edgellm_plugin.so # v0.8.0 plugin the worker links (IPluginV3)
└── models/qwen3-tts-customvoice/
├── engines/talker/
│ ├── llm.engine # FINAL new-calib int4 talker
│ └── config.json # contains codec_language_id + speaker_id maps
├── talker_sidecars/
│ ├── text_embedding_fp8.safetensors # F8_E4M3 [151936,2048] + text_embedding_scales F32 [151936,16]
│ ├── embedding.safetensors # fp16
│ └── text_projection.safetensors # fp16
├── tokenizer/ # CustomVoice talker tokenizer (assistant role NON-EMPTY)
│ ├── tokenizer.json
│ ├── tokenizer_config.json
│ └── processed_chat_template.json
├── code_predictor/ # v0.8.0 highperf-v080 cp_dir
│ ├── llm.engine
│ ├── codec_embeddings.safetensors
│ ├── lm_heads.safetensors
│ └── config.json
└── code2wav/ # v0.8.0 highperf-v080 vocoder
├── code2wav.engine
└── config.json
Deploy: untar, then place text_embedding_fp8.safetensors next to the talker engine as the
talker dir's text_embedding.safetensors (the fp8 file IS the talker's text_embedding tensor),
alongside embedding.safetensors, text_projection.safetensors, config.json, and the tokenizer
files — mirroring the talker dir layout the worker expects.
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/lib/aarch64-linux-gnu:$LD_LIBRARY_PATH
export EDGELLM_PLUGIN_PATH=$PWD/jv-workers/libNvInfer_edgellm_plugin.so
M=$PWD/models/qwen3-tts-customvoice
REQ='{"id":"s1","text":"你好,很高兴见到你。","speaker":"serena","language":"chinese",
"talker_temperature":0.9,"talker_top_k":50,"talker_top_p":1.0,"repetition_penalty":1.05,
"max_audio_length":1024,"output_file":"out.wav"}'
printf '%s\n' "$REQ" | ./jv-workers/qwen3_tts_worker \
--talkerEngineDir="$M/talker_assembled_dir" \
--codePredictorEngineDir="$M/code_predictor" \
--code2wavEngineDir="$M/code2wav" \
--tokenizerDir="$M/tokenizer" \
--debug
--talkerEngineDir must point at the assembled talker dir (engine + fp8 text_embedding renamed to
text_embedding.safetensors + sidecars + config + tokenizer files). Sampling that matched the
working A/B: talker_temperature 0.9, talker_top_k 50, talker_top_p 1.0, repetition_penalty 1.05.
QWEN3_TTS_PRELOAD_TALKER_EMBEDS is deliberately NOT set (real text→speech path, no preload gate).
Available speakers (from engines/talker/config.json speaker_id):
serena (default), aiden, dylan, eric, ono_anna, ryan, sohee, uncle_fu, vivian.
7d3fabe24661a0ce47b03e71b29bd6c5 jv-workers/libNvInfer_edgellm_plugin.so
a9d1e6daa007eb43d5831ad7e1ad1783 jv-workers/qwen3_tts_worker
566c389eea9337195e6b58a846a4ed39 models/qwen3-tts-customvoice/code2wav/code2wav.engine
c6535b47286f069ad04f44378a796e82 models/qwen3-tts-customvoice/code2wav/config.json
0352bb3fe1e228388eae4753f13f8d8f models/qwen3-tts-customvoice/code_predictor/codec_embeddings.safetensors
92271d44d2cf50d829d58da88a9093ea models/qwen3-tts-customvoice/code_predictor/config.json
baff21ea48a9f7e8de30e3d698544e4d models/qwen3-tts-customvoice/code_predictor/llm.engine
63260bf53193b66edffdf8506a4ae2e6 models/qwen3-tts-customvoice/code_predictor/lm_heads.safetensors
7e830c94d6a5149bcd2d1c9829ffab85 models/qwen3-tts-customvoice/engines/talker/config.json
dc5696dc90deb8f8aa29f90ffea969c1 models/qwen3-tts-customvoice/engines/talker/llm.engine
2632299988589ad4d96e151286a4902e models/qwen3-tts-customvoice/talker_sidecars/embedding.safetensors
951bf357184bcd2b702c74db160f84db models/qwen3-tts-customvoice/talker_sidecars/text_embedding_fp8.safetensors
64ca22bd6e83bb8c1d242b3e226dd021 models/qwen3-tts-customvoice/talker_sidecars/text_projection.safetensors
ccee7274ed5bf10a9912253f4ff281a5 models/qwen3-tts-customvoice/tokenizer/processed_chat_template.json
dfd11c77aa705ae64af9c07cd996c233 models/qwen3-tts-customvoice/tokenizer/tokenizer_config.json
5a307103ca203cc774119502276c70cb models/qwen3-tts-customvoice/tokenizer/tokenizer.json
amax/448); produces text_embedding (F8_E4M3
[151936, 2048]) + text_embedding_scales (F32 [151936, 16]).wip/native-int4-talker.873ca22 — fp8 text-embedding wiring in the TTS worker and the
N <= 0 prefill guard. Rebuilt worker md5 a9d1e6da; links v0.8.0 plugin md5 7d3fabe2.Built on Orin NX, TensorRT-Edge-LLM v0.8.0. CustomVoice talker, int4-AWQ + fp8 text-embedding.