fish-s2-pro-ptbr-lora

Fish Audio S2 Pro — Brazilian Portuguese LoRA (Fast AR only)

Built with Fish Audio.

Two LoRA adapters that adapt fishaudio/s2-pro to Brazilian Portuguese. Everything interesting about the model — the 36-layer Slow AR transformer, the codec, the tokenizer, the multilingual ability — is Fish Audio's, unmodified. What is here is a 7.1M-parameter layer on the Fast AR transformer, which is roughly 0.16% of the model.

The base weights are not in this repository. Get them from Fish Audio and accept their license there. This repo ships adapters and documentation, nothing else.

Full pipeline, training scripts, evaluation harness and the stage-by-stage write-up: github.com/JoaoZaokk/ptbr-audio-lab

The two adapters

Same recipe, same steps, same hyperparameters. The only difference is how the 2 hours of audio are distributed across speakers.

A note on counting, because the two columns are not the same thing. Common Voice ships a speaker id; TAGARELA does not, so its clips carry no speaker_id and are grouped by diarized source group instead. "CV speakers" counts only the Common Voice side (46 → 256). "Distinct voice groups" is the honest total across both sources (136 → 363). Either way the ratio is about 2.7×, which is the number the experiment turns on.

The second adapter exists because of a mistake worth documenting: our first training ladder used the same 136 voice groups at every rung, from 30 minutes up to 30 hours. Climbing the ladder added minutes per voice, never voices. The question "does more data help?" had never actually been asked — what had been tested was "does more repetition of the same voice help?". The 6-minute cap fixes that, at the cost of a lower ceiling on total hours.

Results

Measured on 216 utterances, 24 speakers from the Common Voice dev split, disjoint from every training pack. (Our earlier voice bank was contaminated — all 8 evaluation speakers were also training speakers, so its speaker-similarity numbers measured memorization, not cloning. Do not trust similarity numbers that do not state their split.)

Honest reading: the adapters are a small improvement in intelligibility and a statistical tie on everything else. Two blind paired listening rounds against the base model came out 6–6 and 4–5 (both p = 1.000). If you were hoping for a dramatic before/after, this is not that. What they demonstrably do not do is damage the base model, which is the part that took the work.

Other languages are intact

Portuguese adaptation is the kind of thing that quietly breaks a multilingual model. It did not here. 40 generations, 3 languages, each with a reference clip in the same language as the target text — this matters, see below:

An earlier version of this test generated English and Spanish using a Portuguese reference clip, and concluded the model had an accent problem. It did not. The reference had the accent. If you evaluate cross-lingual TTS with a mismatched reference, you are measuring the reference.

Why Fast AR only

The obvious thing to do is train the repo's default LoRA preset, which reaches the Slow AR. We did. It produces the best validation loss of any run we have (9.82 against 12.11) and it is the worst model to listen to: −25 dB of energy decay, 8 dB below the base model in level, CER 3.4× the base.

The failure is not truncation and not a language problem. It is an envelope failure — the model speaks the entire text and fades out while doing it. Amplifying the tail by 34 dB reveals the complete sentence. Three separate metrics (duration, CER, spectral rolloff) all read this as "the audio got cut off", and all three were wrong.

So: the Slow AR is frozen here, and we ship the boring adapter that works. If you train S2 Pro yourself, use energy decay as your guard metric. Validation loss will rank your checkpoints backwards.

Recipe

LoRA preset       r_32_alpha_16_fast   (r=32, alpha=16, alpha/r = 0.5)
targets           fast_embeddings, fast_layers.{0..3}.attention.{wqkv,wo},
                  fast_layers.{0..3}.feed_forward.{w1,w2,w3}, fast_output
frozen            all 36 Slow AR layers, codec, tokenizer
optimizer         lr 1e-5, bf16-true
batch             1 x 8 grad accumulation (effective 8)
max_length        1024
steps             3000, validation every 100
dataset           reference-conditioned, prob_ref = 0.9
hardware          1x RTX 3090, ~7 h per run

val/loss: 13.738 → 12.106 (46 spk) and 13.875 → 12.119 (256 spk). Reported for completeness only — see the warning above about ranking by validation loss.

Two bugs worth reporting

1. The repo's training format is not the inference format

The dataset shipped for fine-tuning builds a sequence the model has never seen at generation time. This is measurable without training anything: feed the raw, untouched s2-pro the two formats and compare loss.

The repo format scores worse than chance on the untouched model. Training against it means the only way for loss to fall is for the model to rewrite its own convention, which is exactly as destructive as it sounds — in our runs it produced −22.4 dB of energy decay. Switching to a reference-conditioned dataset that matches generate_long() took that to −5.8 dB, with the base model at −5.4 dB.

Both adapters here were trained in the corrected format.

2. Semantic embeddings are scaled at inference and not at training

In fish_speech/models/text2semantic/llama.py:

# embed()  — training path
x = self.embeddings(inp[:, 0]) + vq_embeds_sum
return x                                    # no scaling

# forward_generate()  — inference path
if self.config.scale_codebook_embeddings:
    x = torch.where(vq_masks_expanded,
                    x / math.sqrt(self.config.num_codebooks + 1), x)

s2-pro ships scale_codebook_embeddings: true and num_codebooks: 10, so inference divides semantic-position embeddings by √11 ≈ 3.3166 and training does not. Every fine-tune trains against embeddings 3.32× larger than the ones it will see when generating.

We did not find this — notmax123/Fish-Audio-S2-Pro-He did, while training Hebrew, after five runs collapsed. Credit where it is due. Their fix took energy decay from 0.07× to 1.02×. It plausibly explains the Slow AR failure above better than our own explanation did, because the Fast AR sits downstream of the Slow stack and never sees the raw embedding scale — which is exactly the pattern we observed and could not account for.

Disclosure: the adapters in this repository were trained with this bug present. They measure fine because Fast-AR-only training is largely insulated from it, not because it was fixed. If you are training the Slow AR, fix this first.

Emotion and prosody tags

Partially working, and we do not yet have a clean model of why. Measured in pt-BR on the base model and on these adapters:

Note that notmax123 measured no tag response at all on the base model in English (plain/whisper/shouting → RMS 0.0655 / 0.0652 / 0.0689). We see a small but real response in pt-BR. Both can be true; their measurement was level-only.

A control experiment with invented tags ([glorp], [zibbe], [thubner]) at 1×/2×/4× repetition was started and never analyzed — it decides whether the model interprets tags or merely counts brackets. Unfinished. Do not assume tags work until this is done.

One thing that did replicate

A reference clip recorded with tags in its own transcript produces longer output than the same reference recorded without them — 13 of 15 matched pairs, sign test p = 0.0074, mean 12.70 s → 13.41 s (+5.6%). The generated text contains no tags in either condition. Only the reference's style differs. Whether this is expressiveness or merely a slower speaking rate is not yet settled; duration is a proxy.

Data

All source audio is 16 kHz mono, which is why spectral rolloff sits around 3.7 kHz. The base model runs at 44.1 kHz; the adapters do not add high-frequency content they were never shown.

No audio is redistributed in this repository.

Usage

Load the base model from Fish Audio, then apply one adapter:

python fish_speech/models/text2semantic/inference.py \
    --checkpoint-path /path/to/s2-pro \
    --lora-checkpoint adapters/ptbr-2h-256spk/lora.ckpt \
    --lora-config r_32_alpha_16_fast \
    --text '<|speaker:0|>Você está cansado de produtos normais?' \
    --compile \
    --output out.wav

The decoder needs codec.pth from the same fishaudio/s2-pro revision as the weights. Mixing revisions will not error — it will just sound wrong.

For voice cloning, pass a reference clip and its exact transcript. The adapters were trained reference-conditioned at prob_ref = 0.9, so they expect one. Match the reference language to the target language — see above for what happens when you do not.

Limitations

• No audio samples in this repository yet. Our internal evaluation set clones a copyrighted cartoon character, which is not something to publish. Samples built from a CC0 Common Voice speaker are planned.
• Trained on 16 kHz sources. Expect no genuine high-frequency detail.
• 3000 steps at effective batch 8. Small, deliberately: this is a controlled comparison of voice distribution, not a maximum-quality run.
• Improvement over the base model is small and, on most metrics, within noise.
• Trained with bug #2 above unfixed.
• Emotion tag behavior is incompletely characterized.

Restrictions

Research and non-commercial use only. This is required twice over: by the Fish Audio Research License on the base model, and by CC BY-NC-SA 4.0 on the TAGARELA training data. Commercial use of the Fish Audio Materials or any Derivative Work requires a separate written agreement with Fish Audio at [email protected].

Inherited from the TAGARELA dataset card, and repeated here because they matter: do not use this for speaker identification of private individuals, biometric verification, surveillance, unauthorized voice cloning, impersonation, or deceptive synthetic media. Clone voices you have permission to clone.

Per the Fish Audio Research License, do not use these models or their outputs to train or improve a foundational generative AI model.

License and attribution

Built with Fish Audio.

This model is licensed under the Fish Audio Research License, Copyright © 39 AI, INC. All Rights Reserved. The full license text is in LICENSE.md; the required modification statement is in NOTICE.

Under Section V of that license, low-rank adaptation models derived from a Fish Audio Model are Derivative Works, so these adapters fall squarely under it.

Acknowledgements

Fish Audio / 39 AI, INC., for S2 Pro. To be completely clear about the division of labour here: they trained a 4.6-billion-parameter multilingual speech model, aligned it with RL, built the codec and the tokenizer, and released it under a license that permits exactly this kind of work at no cost. We trained 7.1 million parameters on two hours of podcast audio on a single consumer GPU, and most of what this repository contributes is a list of ways we misread our own measurements. Built with Fish Audio, and it shows.

• freds0 for the TAGARELA dataset and the ICASSP 2026 paper.
• Mozilla Common Voice contributors, whose CC0 recordings are the only part of the training data that can be redistributed freely.
• notmax123 for finding the embedding-scale bug and writing it up publicly instead of quietly fixing it.
• TachibanaKimika, whose adapter repository is the model this one follows for license handling and base-revision pinning.
• neko-legends, for publishing a serving profile with no weights in it.

Citation

@misc{ptbr_audio_lab_s2pro_lora,
  title  = {Fish Audio S2 Pro — Brazilian Portuguese LoRA (Fast AR only)},
  author = {JoaoZaokk},
  year   = {2026},
  note   = {Built with Fish Audio. Adapters for fishaudio/s2-pro,
            distributed under the Fish Audio Research License.},
  url    = {https://huggingface.co/JoaoZaokk/fish-s2-pro-ptbr-lora}
}

Please also cite the base model and the training data:

@misc{fishaudio_s2pro,
  title  = {Fish Audio S2 Pro},
  author = {Fish Audio / 39 AI, INC.},
  url    = {https://huggingface.co/fishaudio/s2-pro}
}

@inproceedings{tagarela2026,
  title     = {TAGARELA: A Portuguese Speech Dataset From Podcasts},
  booktitle = {ICASSP},
  year      = {2026},
  url       = {https://huggingface.co/datasets/freds0/TAGARELA}
}