F5-Hindi-onnx-fp16

F5-HINDI ONNX TTS Runtime

Local Hindi text-to-speech using an FP16 F5-TTS transformer, ONNX Runtime CUDA, and a short male reference recording. This build is working on a 4 GB GTX 1650 and has produced a continuous 420-word Hindi sample without reducing the 16-step flow schedule.

Current status

• Working: short Hindi synthesis on CUDA.
• Working: long-form generation through sentence-aware chunks.
• Working: deterministic seeds, resume-safe chunk files, ordered assembly, and background PCM writing.
• Working: 24 kHz, mono, 16-bit PCM WAV output.
• Not real-time on the tested GTX 1650: measured RTF is approximately 3.5.
• Not included: INT8, Q12, GGUF, reduced steps, or another quality-reducing quantization path.
• Experimental and excluded: CUDA Graph and TensorRT backends; see Failed and rejected optimizations.

The inference driver is ../test_f5_hindi_onnx.py. The Colab audiobook workflow is ../F5_Hindi_ONNX_Audiobook_Colab.ipynb.

Why this build is useful

• It runs locally after the Python/CUDA dependencies are available.
• Inference needs ONNX Runtime and NumPy, not the original PyTorch training project.
• The four ONNX graphs total about 382 MiB. The current folder is about 390 MiB including reference material and generated test outputs.
• The transformer remains FP16 and uses all 16 exported flow steps.
• A 3–12 second reference recording controls the speaker identity.
• Text is validated before inference. Unsupported characters fail clearly instead of being silently discarded.
• ONNX Runtime graph optimization is set to ORT_ENABLE_ALL.
• I/O binding and persistent GPU buffers avoid unnecessary full tensor copies between flow steps.
• Long jobs are recoverable. Each completed chunk is written atomically before the next chunk finishes.
• Long chunks are generated from largest to smallest to reduce CUDA arena fragmentation, then assembled back into story order.
• PCM conversion and disk writing overlap GPU inference without running two competing model generations on the 4 GB GPU.

Tested hardware and runtime

These measurements are specific to this machine and should not be treated as universal model benchmarks.

Files

Do not rename or remove the four ONNX graphs, vocab.txt, Sample.wav, or Sample.txt unless the paths in the driver are updated too.

Quick start

Run commands from the repository root, one directory above this folder.

Requirements

• Linux with a working NVIDIA driver and CUDA-capable GPU.
• uv for the tested isolated runtime command.
• Enough free VRAM for the selected chunk. The tested minimum is 4 GiB with the memory strategy in the current driver.

Short test

UV_CACHE_DIR=/tmp/f5-onnx-uv-cache \
UV_TOOL_DIR=/tmp/f5-onnx-uv-tools \
UV_PYTHON_INSTALL_DIR=/tmp/f5-onnx-uv-python \
uv run --with 'onnxruntime-gpu[cuda,cudnn]>=1.20' \
  test_f5_hindi_onnx.py

Expected final line:

TEST_OK provider=CUDAExecutionProvider ... output=.../F5-HINDI/test-output.wav

Reproduce the 420-word test

F5_TEST_WORDS=420 \
F5_CHUNK_CHARACTERS=160 \
UV_CACHE_DIR=/tmp/f5-onnx-uv-cache \
UV_TOOL_DIR=/tmp/f5-onnx-uv-tools \
UV_PYTHON_INSTALL_DIR=/tmp/f5-onnx-uv-python \
uv run --with 'onnxruntime-gpu[cuda,cudnn]>=1.20' \
  test_f5_hindi_onnx.py

This command reads part-1/output/Shadow_slave ep1.txt, uses its first 420 story words, transliterates Sunny to सनी, creates 17 chunks, and writes:

• F5-HINDI/test-output-420-words.wav
• F5-HINDI/test-output-420-words.txt

To prepare and validate only the transcript without loading CUDA sessions:

F5_TEST_WORDS=420 \
F5_TRANSCRIPT_ONLY=1 \
UV_CACHE_DIR=/tmp/f5-onnx-uv-cache \
UV_TOOL_DIR=/tmp/f5-onnx-uv-tools \
UV_PYTHON_INSTALL_DIR=/tmp/f5-onnx-uv-python \
uv run --with 'onnxruntime-gpu[cuda,cudnn]>=1.20' \
  test_f5_hindi_onnx.py

Benchmarks

RTF is wall-clock generation time divided by produced audio duration. An RTF of 1.0 is real-time; lower is faster. These are single-run measurements, not multi-run statistical benchmarks.

The final 420-word output contains approximately 2 minutes 36 seconds of audio and required approximately 9 minutes 21 seconds to generate. Its duration includes a 180 ms pause appended to every chunk.

The CUDA Graph waveform differed from the CUDA baseline by less than one 16-bit PCM step at its peak (difference peak about -90.3 dBFS). Quality was effectively preserved, but runtime became worse because graph-capture overhead dominated the small number of replays.

Long-form generation design

One 400-word call is not valid for this export. Preprocessing estimates the required sequence length, and each ONNX call is limited to max_duration <= 4096. Long-form text therefore uses this pipeline:

1. Normalize punctuation and selected names.
2. Validate every character against vocab.txt.
3. Split text near sentence boundaries, with a configurable character limit.
4. Sort chunks by encoded length from largest to smallest for generation.
5. Seed each chunk using its original story index.
6. Run one CUDA inference at a time with the full 16 steps.
7. Write completed chunks atomically in a background thread.
8. Resume from any valid chunk after interruption.
9. Assemble chunk files back into the original story order.
10. Delete the temporary chunk cache after successful assembly.

Generating out of order does not reorder the audiobook: chunk filenames retain the original story index, and assembly always reads them in story order.

Memory issue and fix

The first 420-word attempt generated eight chunks and then failed at story chunk 9 with:

Failed to allocate memory for requested buffer of size 304166400

The chunk itself was not too large. On a clean retry it generated successfully. The failure came from variable-length allocations fragmenting the 4 GiB CUDA arena.

The working configuration uses both of these measures:

("CUDAExecutionProvider", {"arena_extend_strategy": "kSameAsRequested"})

• Grow the arena by the exact requested size rather than by the next power of two.
• Generate the largest chunks first so later, smaller tensors can reuse the already allocated blocks.

This retry completed all 17 chunks without reducing precision or flow steps.

Known issues and limitations

Performance

• The tested GTX 1650 is approximately 3.5 times slower than real-time.
• I/O binding reduces unnecessary transfers but did not materially improve the measured runtime because transformer compute is the main bottleneck.
• Shorter chunks reduce peak memory but add more preprocessing, decoding, and boundary overhead.
• Longer chunks may sound more continuous but can exceed the 4096-frame limit or 4 GiB VRAM.
• The log reports one CUDA Memcpy node. It may have a small negative effect, but removing it has not yet been validated.
• Two simultaneous GPU generations are intentionally not used. They would compete for the same 4 GiB VRAM and may be slower or fail with OOM.

Audio quality and continuity

• Speaker similarity depends strongly on a clean reference recording and an exact reference transcript.
• The reference must be mono-compatible and between 3 and 12 seconds after conversion to 24 kHz.
• Independent chunk seeds can introduce small changes in energy or delivery at boundaries.
• The 180 ms pause hides abrupt joins but may sound too long or too short for some punctuation.
• RMS validation detects silence or broken output; it does not prove correct pronunciation or emotional delivery.
• No ASR round-trip validation is currently run on every generated chunk.

Driver scope

• The local test driver is tied to part-1/output/Shadow_slave ep1.txt.
• Its built-in Latin-name replacement currently covers only Sunny.
• The Colab notebook contains a larger Shadow Slave pronunciation map, but that map is still not a general Hindi transliteration system.
• Changing the reference voice requires changing both Sample.wav and the exact Sample.txt transcript.
• The export is designed and tested for 16 flow steps. Merely changing the Python constant is not a validated 12-step or Q12 conversion.

Vocabulary issues and future improvement

vocab.txt is a character vocabulary with 105 entries. It covers a practical subset of Devanagari, common Hindi matras, nukta, ASCII digits, basic ASCII punctuation, and the danda ।. It is small and predictable, but it is the main language-coverage limitation.

Current vocabulary problems

• Latin letters are absent. Names such as Sunny, Shadow, acronyms, URLs, and English technical terms must be transliterated before inference.
• Devanagari digits ०१२३४५६७८९ are absent even though ASCII digits 0-9 are present.
• Typographic punctuation such as curly quotes, en/em dashes, and ellipsis is absent. The driver normalizes several of these to ASCII equivalents.
• Brackets, braces, underscore, backslash, pipe, and several other ASCII symbols are absent.
• Emoji, currency signs outside the small ASCII set, and non-Devanagari Indian scripts are unsupported.
• Invisible Unicode characters such as zero-width joiners/non-joiners can cause validation errors unless normalized or removed deliberately.
• The character ङ is absent. The current driver changes it to anusvara ं so the sample transcript works. That is acceptable for this tested name but is not a linguistically correct universal substitution, especially for Sanskrit text.
• Less common Devanagari/Sanskrit characters such as ऽ, ॥, ॐ, ळ, and Vedic marks are absent.
• Character tokenization does not solve pronunciation ambiguity. Different spellings, abbreviations, numbers, and English names still need text normalization or a Hindi grapheme-to-phoneme layer.

The driver intentionally raises Unsupported characters before inference. Do not replace this with silent character deletion; silent deletion can change names, numbers, and meaning without making the failure visible.

Important: do not only edit vocab.txt

Adding a new line to vocab.txt does not teach the model the sound of that character. A new vocabulary ID also needs a corresponding trained text embedding. Changing only the text file can cause an embedding-index error or untrained, unpredictable pronunciation.

Safe vocabulary expansion requires:

1. Define and version a canonical Unicode normalization policy.
2. Build a representative Hindi corpus and inventory every required character.
3. Expand the tokenizer/vocabulary and the model text embedding together.
4. Initialize new embeddings deliberately.
5. Fine-tune the model on recordings containing the new symbols and contexts.
6. Re-export all matching ONNX graphs and vocabulary as one versioned package.
7. Run pronunciation and speaker-similarity regression tests before release.

Recommended improvement order

1. Text normalization first: normalize Unicode, punctuation, whitespace, Hindi/ASCII numerals, dates, money, abbreviations, and common symbols.
2. Transliteration second: add a configurable dictionary plus a proper English-to-Devanagari fallback for unknown Latin words.
3. Pronunciation tests: maintain a small Hindi suite covering conjuncts, nukta letters, anusvara/chandrabindu, names, numbers, and punctuation.
4. Vocabulary fine-tune: add truly missing phonetic characters only when normalization and transliteration cannot represent them correctly.
5. ASR validation: transcribe generated regression clips and flag dropped, repeated, or substituted words.

Useful future regression examples should include:

• कंगन, गंगा, अँगना, and शङ्कर to distinguish nasal forms.
• क़िला, ख़बर, ग़ज़ल, ज़मीन, फ़र्क़, ड़, and ढ़ for nukta.
• Hindi and ASCII numerals, dates, percentages, currency, and decimals.
• English character names and Shadow Slave terms after transliteration.
• Long conjuncts such as क्ष, त्र, ज्ञ, and श्र.

Failed and rejected optimizations

CUDA Graphs

CUDA Graph capture kept the output numerically equivalent at 16-bit playback precision but made the short test slower: 44.01 seconds instead of about 35.3 seconds. Dynamic sequence shapes and capture overhead make it a poor default for this workload. It should only be revisited with fixed-shape buckets and multiple repeated calls per captured shape.

TensorRT

ONNX Runtime 1.28 expected TensorRT 10 libraries. A matching TensorRT 10.16 CUDA runtime was tested, but its library wheel exceeded 3 GiB, the temporary runtime expanded to about 20 GiB, and the optimization-level-5 engine build was interrupted before an engine cache was serialized. No TensorRT speed result was obtained.

The temporary project cache and incomplete download were removed. TensorRT is not required by the working CUDA path. Revisit it only on a machine with ample disk/RAM and enough time for a full cold engine build, then compare warm runtime and audio before keeping it.

Quantization and fewer steps

• INT8 was not used because it can change voice quality and requires proper calibration.
• Q12 is not a standard ONNX Runtime quantization format for this pipeline.
• GGUF was not produced; the working F5 pipeline is a multi-graph flow model, not a single LLM graph.
• Reducing the 16 flow steps was rejected for this quality-preserving build.

Recommended settings for this GPU

Troubleshooting

CUDAExecutionProvider is unavailable

Confirm that an NVIDIA GPU is visible and run with the GPU package, not the CPU package:

nvidia-smi

The driver calls ort.preload_dlls() so CUDA and cuDNN libraries installed by the ONNX Runtime package can be loaded before sessions are created.

Unsupported characters

Do not remove the assertion. Normalize or transliterate every reported symbol, then run transcript-only mode again. If the character is genuinely required, follow the vocabulary fine-tuning process rather than editing vocab.txt alone.

max_duration exceeds 4096

Reduce F5_CHUNK_CHARACTERS. Start with 140 or 120 on a difficult passage. Byte length and reference-text length affect the estimate, so equal character counts do not always create equal duration.

CUDA allocation failure

• Keep arena_extend_strategy=kSameAsRequested.
• Generate largest chunks first.
• Stop other GPU applications.
• Reduce F5_CHUNK_CHARACTERS if the largest chunk itself fails on a clean process.
• Rerun the same command; valid cached chunks resume automatically.

Output sounds wrong but passes RMS validation

Check, in order:

1. Sample.txt exactly matches what is spoken in Sample.wav.
2. The reference is clean, single-speaker speech without music or reverb.
3. The input contains no unexpanded English names, abbreviations, or numbers.
4. Chunk boundaries do not split a name or sentence unnaturally.
5. The source text uses the intended Hindi spelling.

Model provenance and redistribution

The original Hindi checkpoint used for this ONNX package came from SPRINGLab/F5-Hindi-24KHz, including model_2500000.safetensors / model_2500000.pt. The original checkpoint, Vocos files, conversion cache, and training configuration are not kept in this cleaned runtime folder; they are redownloadable from the model source.

Before redistributing the ONNX files or generated voices, review the current model license and the reference speaker's consent/usage rights. Use only voices you own or have permission to clone.