Finatts-enhanced

<div align="center">

🔊 Finatts Enhanced

High-fidelity voice cloning — fine-tuned Spark-TTS v2

Text-to-Speech · Voice Cloning · Emotion Tags · Portable Voice Profile

Built by Featherlabs · Operated by Owlkun

</div>

---

✨ What is Finatts Enhanced?

Finatts Enhanced is an improved 507M-parameter text-to-speech model built on Spark-TTS-0.5B, fine-tuned for high-fidelity single-speaker voice cloning with emotion tag support.

Compared to the original Finatts, this version features 3× the training, a more stable learning rate, and a portable voice profile (elise_voice.safetensors) — no reference audio needed at inference time.

Improvements over v1

🎯 Built For

---

🏋️ Training Details

<table> <tr><td><b>Property</b></td><td><b>Value</b></td></tr> <tr><td>Base model</td><td><a href="https://huggingface.co/SparkAudio/Spark-TTS-0.5B">SparkAudio/Spark-TTS-0.5B</a></td></tr> <tr><td>LLM backbone</td><td>Qwen2-0.5B (507M params)</td></tr> <tr><td>Dataset</td><td><a href="https://huggingface.co/datasets/MrDragonFox/Elise">MrDragonFox/Elise</a> (1,195 samples, ~3h)</td></tr> <tr><td>Training type</td><td>Full Supervised Fine-Tuning (SFT)</td></tr> <tr><td>Epochs</td><td>6</td></tr> <tr><td>Batch size</td><td>8 (effective 16 with grad accum)</td></tr> <tr><td>Learning rate</td><td>5e-5</td></tr> <tr><td>Warmup steps</td><td>50</td></tr> <tr><td>Weight decay</td><td>0.01</td></tr> <tr><td>Context length</td><td>4,096 tokens</td></tr> <tr><td>Precision</td><td>BF16</td></tr> <tr><td>Optimizer</td><td>AdamW (torch fused)</td></tr> <tr><td>LR scheduler</td><td>Cosine</td></tr> <tr><td>Framework</td><td>Unsloth + TRL (SFTTrainer)</td></tr> <tr><td>Hardware</td><td>AMD MI300X (192GB HBM3)</td></tr> </table>

📊 Training Metrics

Training Loss Curve

Model converges from ~6.9 → ~5.8 over 450 steps — 3× more convergence than v1:

---

🚀 Quick Start

Prerequisites

pip install "unsloth[amd] @ git+https://github.com/unslothai/unsloth"
pip install "transformers<=5.2.0,>=4.51.3" "trl<=0.24.0,>=0.18.2"
pip install omegaconf einx "datasets>=3.4.1,<4.4.0" soundfile safetensors

# Clone Spark-TTS for BiCodec tokenizer
git clone https://github.com/SparkAudio/Spark-TTS

Inference with Elise Voice Profile

import torch, re, sys
import soundfile as sf
from transformers import AutoTokenizer, AutoModelForCausalLM
from huggingface_hub import snapshot_download, hf_hub_download
from safetensors.torch import load_file
import json

sys.path.append("Spark-TTS")
from sparktts.models.audio_tokenizer import BiCodecTokenizer

MODEL_ID = "Featherlabs/Finatts-enhanced"

# Load LLM
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto"
)
model.eval()

# Load BiCodec
snapshot_download("unsloth/Spark-TTS-0.5B", local_dir="Spark-TTS-0.5B")
audio_tokenizer = BiCodecTokenizer("Spark-TTS-0.5B", "cuda")

# Load Elise voice profile (global token IDs — no reference audio needed)
profile_path = hf_hub_download(MODEL_ID, "elise_voice_profile.json")
with open(profile_path) as f:
    profile = json.load(f)
elise_global_ids       = profile["global_token_ids"]
elise_global_token_str = profile["global_token_str"]


@torch.inference_mode()
def generate_speech(text, temperature=0.8, top_k=40, top_p=0.92):
    prompt = "".join([
        "<|task_tts|>",
        "<|start_content|>", text, "<|end_content|>",
        "<|start_global_token|>",
        elise_global_token_str,       # Elise's voice injected here
        "<|end_global_token|>",
        "<|start_semantic_token|>",
    ])
    inputs    = tokenizer([prompt], return_tensors="pt").to("cuda")
    generated = model.generate(
        **inputs, max_new_tokens=2048,
        do_sample=True, temperature=temperature,
        top_k=top_k, top_p=top_p,
        eos_token_id=tokenizer.eos_token_id,
    )
    out = tokenizer.batch_decode(
        generated[:, inputs.input_ids.shape[1]:], skip_special_tokens=False
    )[0]
    sem = [int(t) for t in re.findall(r"bicodec_semantic_(\d+)", out)]
    if not sem:
        return None
    pred_sem    = torch.tensor(sem, dtype=torch.long).unsqueeze(0).to("cuda")
    pred_global = torch.tensor(elise_global_ids, dtype=torch.long).unsqueeze(0).to("cuda")
    audio_tokenizer.model.to("cuda")
    return audio_tokenizer.detokenize(pred_global, pred_sem).squeeze().cpu().numpy()


# Try emotion tags!
texts = [
    "Hey there! My name is Elise, nice to meet you.",
    "<laughs> Oh my gosh, I can't believe that actually worked!",
    "<whispers> Come closer... I have a secret to tell you.",
    "<sighs> Some days just feel heavier than others.",
]
for i, text in enumerate(texts):
    wav = generate_speech(text)
    if wav is not None:
        sf.write(f"output_{i+1}.wav", wav, 16000)
        print(f"✅ output_{i+1}.wav")

---

🎭 Emotion Tags

The Elise dataset includes inline emotion tags captured from real speech. Place them anywhere in your text:

Note: Tags produce intonation variation rather than literal acoustic sounds (e.g., actual giggling audio). For acoustic emotion effects, see Orpheus-TTS.

---

🏗️ Architecture

Text + Emotion Tags
        ↓
  [LLM: Qwen2-0.5B]
  ┌─────┴──────┐
  Global tokens   Semantic tokens
  (speaker ID)    (content + prosody)
       └────────┬────────┘
         [BiCodec Decoder]
                ↓
           Waveform 16kHz

---

📦 Repository Files

For inference you also need:

---

⚠️ Limitations

• English only — only tested with English text inputs
• Single speaker — optimized for Elise; base model multi-speaker may be degraded
• 16kHz output — use audiosr for upsampling to 44.1kHz
• Emotion intensity — tags produce subtle intonation changes, not acoustic emotion sounds
• ROCm-trained — tested on AMD MI300X; CUDA users may need minor env adjustments

---

🔮 What's Next

• 🔊 Super-resolution — integrate audiosr for 44.1kHz HD output
• 🗣️ Multi-speaker — train on multiple voices
• 📈 Larger dataset — more hours of Elise audio for stronger emotion control
• 🎭 Acoustic emotions — explore Orpheus-style explicit emotion tokens

---

📜 License

Apache 2.0 — consistent with Spark-TTS-0.5B.

---

<div align="center">

Built with ❤️ by Featherlabs

Operated by Owlkun

</div>