audio8-tts-from-scratch

weights-and-wires/audio8-tts-from-scratch

A DualAR text-to-speech language model trained from scratch on 440 clips (72.6 minutes) of LibriSpeech clean/train.100, 4 speaker(s).

⚠️ This is a training-pipeline experiment, not a production TTS model. The ~601 M-parameter transformer was randomly re-initialized and trained for 280 optimizer steps. Expect undertrained, partly unintelligible speech. LibriSpeech is 16 kHz upsampled to the codec's 44.1 kHz, which caps quality further.

What was trained vs. reused

Training configuration

Final metrics

Random-guess baselines for reference: slow_loss ln(vocab) = 11.96, fast_loss ln(4096) = 8.32.

Usage

import soundfile as sf, torch
from transformers import AutoModel, AutoProcessor

repo = "weights-and-wires/audio8-tts-from-scratch"
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).eval().cuda()

inputs = processor(text=["Hello world."], return_tensors="pt")
inputs = {k: v.cuda() for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=512, temperature=0.8, top_p=0.95,
                     top_k=50, do_sample=True, return_dict_in_generate=True)
audio, lengths = model.decode_audio(out.codes)
sf.write("out.wav", audio[0, :int(lengths[0])].float().cpu().numpy(), 44100)

The model was trained on the no-reference prompt template only — pass text= alone; voice cloning via reference_audio= was never trained and will not work.

License

Apache-2.0, inherited from the base checkpoint. LibriSpeech is CC-BY-4.0.