F5TTS-Hindi

Repository Setup for Data and model files

• Install git-xet on your local machine. This is huggingface's tool to manage large files

  • Mac brew install git-xet
  • Ubuntu curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/huggingface/xet-core/refs/heads/main/git_xet/install.sh | sh

• Install huggingface_cli for authentication

  • Ubuntu curl -LsSf https://hf.co/cli/install.sh | bash

• Login to huggingface using your account credentials hf auth login

• Initialize git xet in the repo git xet install

• Force Git LFS to only use Hugging Face Run this command in your local repo. git config xet.url "https://huggingface.co/datasets/ashish19goyal/F5TTS-Hindi"

• Tell git-xet to manage model and data files git xet track "*.pt" "*.wav" "*.arrow" "*.jsonl" "*.json" "*.png" "*.pdf" "*.safetensors" "*.pth"

• Commit attributes file to git git add .gitattributes git commit -m "Add git-xet attributes"

• Workflow

  • Push code + pointers to GitHub git push origin main

  • Push the actual large data files to Hugging Face git push hf main

Configuration

This setup uses mutliple tools for different purposes as listed below

• Apache airflow: To run pipelines for each of the objectives listed above.
• Apache spark: To run each of the tasks in the pipelines in a distributed manner. This helps in partitioning the data and avoiding out-of-memory errors
• MLFlow: To track multiple training runs with different configurations and model registry
• Git xet: To store processed data and model checkpoints on huggingface.
• docker: To deploy Apache airflow, spark, mlflow clusters
• ray.tune: For hyperparameter tuning

Additional setup

• Create a variables.json file in the root of the repo with following content. This file is not committed to prevent secrets from being exposed.
• If you use Gmail SMTP, do not use your normal Google account password. Use a Google App Password (requires 2-Step Verification) from https://myaccount.google.com/apppasswords.

{
    "hf_token": "<huggingface_token>",
    "smtp_username": "<smtp_username>",
    "smtp_password": "<smtp_app_password>",
    "notify_email_to": "<recipient_email>"
}

Serve

Serve runs on Kubernetes (minikube). Images are built on host Docker (BuildKit cache in .docker-cache/) and then loaded into minikube — this avoids rebuilding inside the minikube VM (which often hung on “exporting layers” on Windows).

Use Git Bash / WSL / macOS / Linux for run.sh and the kubectl snippets below (PowerShell mangling of JSON patches is awkward; bash is the supported path).

Inference backend: Ray vs Kafka

INFERENCE_BACKEND selects how FastAPI (tasks/http-serve.py) submits jobs:

Set the same value on both serve and inference (ConfigMaps in k8s/serve.yaml and k8s/inference.yaml, or kubectl set env):

# Ray (default — already in manifests)
kubectl -n f5-tts set env deployment/serve INFERENCE_BACKEND=ray
kubectl -n f5-tts set env deployment/inference INFERENCE_BACKEND=ray

# Kafka fallback (requires Kafka pods up)
kubectl -n f5-tts set env deployment/serve INFERENCE_BACKEND=kafka
kubectl -n f5-tts set env deployment/inference INFERENCE_BACKEND=kafka

Deploy / rebuild tips

# Full wipe, then first deploy (slow once)
bash run.sh serve-clean
bash run.sh serve

# Later deploys reuse host image + .docker-cache (fast unless you change Dockerfiles)
bash run.sh serve

# Force a full image rebuild
FORCE_BUILD=1 bash run.sh serve

# Manifests only (images already on host)
SKIP_BUILD=1 bash run.sh serve

# Faster smoke deploy: skip kube-prometheus-stack + Grafana ServiceMonitors
SKIP_PROMETHEUS=1 bash run.sh serve

bash run.sh serve-down   # remove k8s apps, keep minikube + images

run.sh serve already scales serve + inference to 0, removes the old minikube image tags, then minikube image load … --overwrite=true before scaling back up. If you reload images by hand, do the same — otherwise Docker/minikube may keep serving a stale :local layer while pods still hold the old image:

kubectl -n f5-tts scale deployment/serve --replicas=0
kubectl -n f5-tts scale deployment/inference --replicas=0
minikube image rm f5-tts-serve:local f5-tts-inference:local 2>/dev/null || true
minikube image load f5-tts-serve:local --overwrite=true
minikube image load f5-tts-inference:local --overwrite=true
kubectl -n f5-tts scale deployment/serve --replicas=1
kubectl -n f5-tts scale deployment/inference --replicas=1

Host-GPU workaround (Docker GPU OK, minikube GPU not)

Use this when docker run --gpus all … nvidia-smi works but the cluster does not advertise nvidia.com/gpu (inference in-cluster stays on CPU / very slow).

Flow: keep serve in minikube, run the Ray worker on the host with GPU, point serve at the host Ray Client port, and retarget the wait-for-ray init container (it otherwise waits forever on in-cluster ray-head while inference is scaled to 0).

1. Prerequisites

# Windows (PowerShell): scripts/check-docker-gpu.ps1
# Linux/macOS:
docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi

Host gateway hostname from a minikube pod (Docker driver): usually host.minikube.internal. If that fails, try host.docker.internal.

2. Stop in-cluster inference; start host Ray worker

kubectl -n f5-tts scale deployment/inference --replicas=0

docker rm -f f5-ray 2>/dev/null || true
docker run --rm -d --gpus all --name f5-ray \
  -p 10001:10001 -p 6379:6379 \
  -e INFERENCE_BACKEND=ray \
  -e RAY_NUM_GPUS=1 \
  f5-tts-inference:local

docker logs -f f5-ray   # wait until Ray worker / actors look ready

3. Point serve at host Ray + fix init wait

HOST_RAY=host.minikube.internal   # or host.docker.internal

kubectl -n f5-tts set env deployment/serve \
  RAY_ADDRESS="ray://${HOST_RAY}:10001"

# Patch wait-for-ray so it probes the host, not ray-head Service
cat >/tmp/serve-init-patch.json <<EOF
[{"op":"replace","path":"/spec/template/spec/initContainers/0/command","value":["/bin/sh","-c","until nc -z ${HOST_RAY} 10001; do echo waiting for ${HOST_RAY}:10001...; sleep 5; done; echo ray host reachable"]}]
EOF
kubectl -n f5-tts patch deployment serve --type=json --patch-file=/tmp/serve-init-patch.json

kubectl -n f5-tts rollout status deployment/serve --timeout=5m

If serve was already running against a dead Ray client, restart after f5-ray is up:

kubectl -n f5-tts rollout restart deployment/serve
kubectl -n f5-tts rollout status deployment/serve --timeout=5m

4. Call the API

NodePort (minikube ip:30800) often fails on Windows Docker Desktop — prefer port-forward on all platforms:

kubectl -n f5-tts port-forward svc/serve 8000:8000

curl http://127.0.0.1:8000/health
curl -X POST "http://127.0.0.1:8000/predict/" \
  -H "Content-Type: application/json" \
  -d '{"text":"मैं बाजार जा रहा हूँ।"}' \
  --output out.wav
ls -lh out.wav

Tiny (~1KB) out.wav is usually a JSON error body, not audio — open it in an editor.

5. Tear down host worker / restore in-cluster Ray

docker stop f5-ray
kubectl -n f5-tts scale deployment/inference --replicas=1
# Re-apply manifests (or set RAY_ADDRESS back to ray://ray-head:10001 and restore wait-for-ray)
SKIP_BUILD=1 bash run.sh serve

Quick health check (in-cluster Ray path)

curl "http://$(minikube ip):30800/health"
# or: kubectl -n f5-tts port-forward svc/serve 8000:8000
curl -X POST "http://127.0.0.1:8000/predict/" -H 'Content-Type: application/json' -d '{"text": "I am going to buy some vegetables from the village. It will take almost a day. I will be back by tomorrow. Dont wait for me. You keep working on your studies in the meanwhile"}' --output out.wav

Usage

Entry points: ./run.sh

Commands:

• Data analytics: ./run.sh analyse
  • It produces the analysis report at data/analysis/report.html
• Data cleanup: ./run.sh cleanup
  • It produces the cleaned data at data/cleanup
• Data preaparation: ./run.sh prepare
  • This produces the vocabulary at data/prepare/text/vocab.txt
  • Audio and text for training are stored as arrow files at data/prepare/arrow
• Hyperparameter tuning: ./run.sh tune
  • This produces the best hyperparameters at data/hyperparameters/best_params.json
• Training: ./run.sh train
• Evaluation: ./run.sh eval
• Serve: ./run.sh serve (see Serve above for Ray/Kafka, host-GPU, timeouts)

VITS2 reference model

hindi_vits2_training.ipynb / kaggle_notebook.py (with the supporting scripts/setup_vits.sh, scripts/train_vits.sh, src/vits_hindi/*) train a VITS2-based Hindi TTS model as a comparison point against the F5-TTS architecture used elsewhere in this repo. It's a separate, self-contained experiment — not part of the Airflow/Spark pipeline above, and not required for it. Run it locally or on Kaggle (vits2/ clone directory is intentionally git-ignored; see .gitignore).

Checkpoints

checkpoints/hindi_vits2/ holds the VITS2 generator/discriminator weights (G_<step>.pth / D_<step>.pth) from the notebook's training runs. Only the 2 most recent steps are kept in the repo at a time — each new checkpoint commit drops the oldest pair to avoid unbounded LFS growth from a model that checkpoints every few thousand steps. "Most recent" is judged by when the checkpoint file was actually produced, not by step number (a later session can resume from and re-save an earlier step). Currently: step 26000 and step 33000 (latest).

Visualizing performance profiles

Use prefetto to visualize the training profile dumps generated in data/artifacts/*_profile.json

• Open https://ui.perfetto.dev in chrome
• Click Open trace file and pick the profile dump for visualizing

Visualize metrics

kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090
# then open http://localhost:9090/targets and look for the f5-tts entries

kubectl -n monitoring port-forward svc/prometheus-grafana 3000:80
# admin / prom-operator