Vocetta2-147k

Vocetta2 147K

A complete English text-to-speech system in 147,532 parameters — three tiny networks and a dictionary-based grapheme-to-phoneme front end, all running on CPU in real time.

Listen to samples/ first — those eight files were rendered by this exact checkpoint.

How it works

Text becomes audio through four stages, all trained by distillation from a larger teacher TTS:

text ──► G2P ──► phoneme ids
                  │
      duration.pt │  ids ──────────────► frame count per phoneme
                  │
      acoustic.pt │  ids + durations ───► mel spectrogram [100 bands, T frames]
                  │
      decoder.pt  │  mel + noise ───────► complex spectrum ──► iSTFT ──► audio

G2P (microtts/g2p/). A dictionary-first English front end: words are looked up in bundled pronunciation dictionaries (gold and silver), and words the dictionaries miss go through a small bundled neural fallback model. NumPy only, no espeak, no torch, no network access. The output is a string of IPA phonemes, mapped to a frozen 62-symbol vocabulary (<bos> and <eos> bracket each utterance).

Duration student (duration.pt, 5,344 params). A 3-layer 1D convolutional network over the phoneme sequence. It predicts how many mel frames each phoneme occupies, using learned position, sequence-length and duration features, with residual blocks around each conv pair. The output is exponentiated log-duration, rounded and clamped to at least one frame per phoneme.

Acoustic student (acoustic.pt, 40,162 params). Embeds the phoneme ids, refines them with token-context convolutions, expands them to the frame grid by repeating each phoneme its predicted number of frames, then runs a second stack of convolutions over frames and projects to 100 mel bands.

Decoder (decoder.pt, 102,026 params). Mel to waveform. A ConvNeXt1D stack (depthwise conv, LayerNorm, two pointwise layers with a GELU between them, residual) maps the mel to a complex spectrum of 513 bins, which the iSTFT turns into audio. The magnitude head is exponential with bin 0 and the Nyquist bin zeroed, and a DC-blocking filter removes the remaining offset. The decoder is noise-fed: a 4-channel noise input is projected and added to the mel embedding. At inference, zero noise is the best choice.

The budget choice

This model exists to answer one question: how much quality survives under 150,000 parameters?

Two allocations were measured at almost exactly the same total size. The one shipped here puts the money in the decoder:

The wider decoder buys naturalness (+0.16 SCOREQ); the wider acoustic buys a little intelligibility (−0.005 WER). Decoder width dominated every capacity comparison made while building this family, which is why the budget went there. A ~276K model trained the same way reaches SCOREQ 2.04 — the extra ~130K is almost entirely decoder, and that is the honest cost of the remaining gap.

The training recipe

Every student is trained by distillation: a larger teacher TTS renders a text corpus once, and the students learn to reproduce the teacher's intermediate representations.

Stage 0 — build the pack

Pick a teacher TTS and a text corpus (thousands of sentences of varied, spoken-style text). For each line store: phoneme ids, teacher audio, per-phoneme durations, and the mel spectrogram of the teacher audio (100 bands, n_fft 1024, hop 256). One .npz per line (train/build_pack.py). Watch the duration units when the teacher's frame rate differs from the mel hop — the script shows the conversion.

Stage 1 — duration student

Train ids → frame counts against the teacher's durations. Loss: smooth-L1 on log-duration plus a term on the total length (weight 0.35). Learning rate 2e-3 with AdamW, ~4k steps at batch 32. Capacity matters and not monotonically: hidden 12 is the sweet spot this model uses; sweep a few sizes if you change the corpus.

Stage 2 — acoustic student

Train ids + durations → mel against the teacher's mel. Loss: L1 plus normalized L1, temporal-delta L1 (weight 0.10 — this anti-smoothing term is load-bearing), channel-statistics L1, and a hinge-loss PatchGAN critic on the mel from step 1500 with weight 0.1. The learning rate matters most: 2e-3, constant. ~30k steps at batch 8.

An extra step is worth naming: this acoustic was distilled against the mel output of a larger acoustic in the same family, not against the teacher's mel directly. Distilling against a reachable target (a function the student can actually represent) was worth more than any capacity change tried here.

Stage 3 — decoder

1. Initialize, do not train from scratch. Slice the first N channels of a pretrained neural vocoder's decoder (train/init_decoder.py; dim 40, pw 120, 3 blocks for this model). Training a decoder this small from scratch does not reach intelligibility.
2. Recovery. Fine-tune the slice on teacher mels (--mix-prob 0.0) with waveform L1 + multi-resolution spectral loss + a hinge PatchGAN discriminator + a cosine-gram temporal-structure loss (weight 0.4 — this supervises temporal texture and is what keeps the output from sounding mushy), plus high-band-excess and quiet-ceiling terms. ~20k steps at batch 4, constant lr 2e-4.
3. Z-mix. Continue with a 50/50 mix of teacher mels and acoustic-student mels (--mix-prob 0.5), another ~20k steps. Load-bearing for the handoff.
4. Joint fine-tune. Co-train the acoustic and decoder together end-to-end against audio targets for ~30k steps. This closes the acoustic→decoder mismatch that no separately-trained pair fully avoids. It must update BOTH networks — the joint stage changes the acoustic substantially, and evaluating the pre-joint acoustic understates the model by ~0.27 SCOREQ.

Two traps measured during development, worth stating plainly:

• Use a constant learning rate. Decaying the rate (e.g. ×0.9998 per step) silently starves these small models: the loss looks fine while the output stays muddy. Removing the decay was the single largest fix in this model's history.
• Do not over-trust one metric. A decoder tuned only against SCOREQ once learned to emit muffled, unintelligible audio that scored 3.07 SCOREQ (WER 1.0). Every candidate here was checked with WER, DNSMOS and listening.

Use it from scratch

Download this repository, then install the runtime dependencies and run the script below. Two commands do the download — pick whichever tool you have:

# either the Hugging Face CLI (install it if you don't have it)
pip install -U huggingface_hub
hf download VantoraLabs/Vocetta2-147k --local-dir vocetta2-147k

# or git-lfs (install git-lfs first: https://git-lfs.com)
git lfs install && git clone https://huggingface.co/VantoraLabs/Vocetta2-147k

cd vocetta2-147k
pip install -r requirements.txt          # numpy, torch, soundfile

Then, from inside that folder:

from microtts import MicroTTS

tts = MicroTTS.load(".")                  # reads duration.pt / acoustic.pt / decoder.pt
wav = tts.synthesize("Hello world.")      # float32 numpy array, 24 kHz
tts.save("out.wav", wav)                  # saves RMS-normalized to -26 dBFS

MicroTTS.load accepts device="cpu" (default) or "cuda". The full pipeline loads in ~0.1 s and needs no network access. If you already have phoneme ids, call tts.synthesize_ids(ids) directly and skip the G2P.

Two practical notes:

• Loudness. synthesize returns the raw output; its loudness is not normalized. MicroTTS.normalize(wav) applies RMS normalization (0.05, about -26 dBFS), and MicroTTS.save does it for you.
• Noise. synthesize(..., noise_scale=0.0) is the default and gives the best measured quality. The decoder still accepts noise_scale=1.0 if you want variation across renders, but it costs quality on every metric.

Python 3.10+, CPU is enough.

Benchmarks

Measured on this exact checkpoint. All word-error numbers use Whisper small as the judge; naturalness scores use the standard open models (SCOREQ, DNSMOS), each evaluated on the raw rendered audio.

Intelligibility (word error rate, lower is better)

The templated set (near-identical sentence frames) is fully memorized; the diverse set is the honest generalisation measure and the number to compare against other models.

Naturalness / audio quality (higher is better, 24-sentence diverse set)

DNSMOS-SIG catches metallic distortion; 3.19 at zero noise says the output is not buzzy.

Speed (24 diverse sentences, warm-up excluded, zero noise, CPU)

RTF includes the duration, acoustic and decoder forwards; the G2P adds ~1 ms per sentence on top.

Reproducing the scores. Word error: transcribe the rendered wavs with openai/whisper-small and compute WER against the input text (scripts in benchmark/). Naturalness: pip install scoreq speechmos, then score each wav with the library's own defaults.

What is in this folder

duration.pt, acoustic.pt, decoder.pt   the weights (606 KB total, fp32)
model.safetensors                       same weights, prefixed keys, fp32
                                        (auto-detected by HF Hub so the params
                                        count shows on the repo card)
microtts/                              runtime package (frontend, models, g2p)
  g2p/g2p_data/                        dictionaries + fallback model, Apache-2.0
samples/                               eight rendered examples
train/                                 the training scripts (see the recipe above)
benchmark/                             RTF + WER measurement scripts and results
README.md, LICENSE, requirements.txt

Limits, stated plainly

• English only. The G2P dictionaries are US English.
• One voice. This is a single-voice model; there is no speaker conditioning.
• Utterances cap at ~207 phoneme tokens and 2400 mel frames (~25 s).
• Free-form conversational text is harder than templated text; word errors are common there (see the benchmark table).
• Naturalness is the price of the 147K budget: intelligible and clean, but audibly less natural than a ~276K model of the same family. The gap is decoder width, and there is no configuration under 150K that closes it.
• No text normalization beyond the front end's number handling. Unusual punctuation or markup should be stripped before synthesis.

Credits

Runtime, weights and training scripts: MIT (this release). The bundled grapheme-to-phoneme dictionaries and the fallback model are Apache-2.0; see microtts/g2p/g2p_data/NOTICE.md.