plapre-nano-v2

plapre-nano-v2

Danish multi-task text-to-speech. A 335M LlamaForCausalLM that autoregressively predicts Kanade 25 Hz audio tokens from Danish BPE text and decodes them to 24 kHz speech. A 128-d speaker embedding — extracted from any reference clip with the Kanade encoder — is projected and prepended to the sequence, so every generation is voiced.

What it can do

Two tasks:

Four controls that stack onto generate:

Combining them

Controls are composable — any subset stacks in one prompt, in this block order:

[SPK]  [pronunciation]  [voice-reference]  [context]  <text> BPE  [pace]  <audio> …

Voice-reference is the only control that swaps the speaker embedding (to the reference clip's); the others keep the target voice. Edit composes with pronunciation only. All combinations are trained, not emergent.

How to use it

The easiest path is the plapre library, which wraps every task and combination:

from plapre import Plapre

tts = Plapre("syvai/plapre-nano-v2")

# plain TTS
tts.speak("Hej, hvordan har du det?", output="out.wav", split_sentences=True)

# clone a voice from any clip
tts.clone("Denne sætning har stemmen aldrig sagt.", reference_wav="voice.wav")

# cloned voice + set pace + pinned pronunciation, in one call
tts.clone(
    "Mette Frederiksen mødte Volodymyr Zelenskyj i København.",
    reference_wav="voice.wav",
    durations=[14, 22, 12, 24, 4, 18],              # one frame count per word
    pronunciations=[("Zelenskyj", "zelenskyj_ref.wav")],
)

# continue a conversation with matching prosody
tts.continue_context("Og det er derfor, vi handler nu.",
                     prev_text="Situationen har ændret sig markant.",
                     prev_wav="previous_line.wav", speaker_wav="voice.wav")

# edit a recording: replace words in place
tts.edit("… ny formulering her …", mask_start=40, mask_end=55,
         original_wav="clip.wav")

Manual inference (transformers)

import numpy as np, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import hf_hub_download

CKPT = "syvai/plapre-nano-v2"
tok = AutoTokenizer.from_pretrained(CKPT)
m = AutoModelForCausalLM.from_pretrained(CKPT, torch_dtype=torch.float32).eval()
spj = torch.nn.Linear(128, m.config.hidden_size)
spj.load_state_dict(torch.load(hf_hub_download(CKPT, "speaker_proj.pt"), map_location="cpu"))
spj.eval()

g = tok.convert_tokens_to_ids
STOPS = [g("</audio>"), tok.eos_token_id]           # stop on BOTH terminators
pre = [g("<text>")] + tok.encode(text, add_special_tokens=False) + [g("<audio>")]

pe = m.get_input_embeddings()(torch.tensor(pre))
spk = spj(torch.tensor(np.asarray(speaker_embedding), dtype=torch.float32)).unsqueeze(0)
inp = torch.cat([spk, pe], 0).unsqueeze(0)
out = m.generate(inputs_embeds=inp,
                 attention_mask=torch.ones(inp.shape[:2], dtype=torch.long),
                 max_new_tokens=500, do_sample=True, temperature=0.7,
                 top_p=0.95, top_k=50, eos_token_id=STOPS,
                 pad_token_id=tok.eos_token_id)[0].tolist()
audio_base = g("<audio_0>")
content = []
for t in out:
    if audio_base <= t < audio_base + 12800:
        content.append(t - audio_base)
    elif content:
        break
# decode `content` with kanade_tokenizer (frothywater/kanade-25hz-clean) -> 24 kHz wav

Control-block prompt formats (reference audio caps, <dur_j> ids, edit masking/splicing) are implemented in plapre/tasks.py — pure token-layout builders you can read or reuse directly.

Inference requirements

• Stop on both terminators: generated audio ends </audio> then <eos> — pass both ids as stop tokens (the library does this for you).
• Run in fp32 (the training precision). Lower precision measurably increases end-of-utterance artifacts.
• Use this repo's tokenizer (vocab 21224).
• End sentences with terminal punctuation — append a "." if your text ends on a comma or nothing; the stop signal is strongest on sentence-final text.
• Split long text into sentences and generate them as a batch; join with ~250 ms of silence. Speaking pace follows the voice: reference clips from calm speakers yield calm narration.
• If serving with vLLM, use vllm>=0.15,<0.16 with enable_prompt_embeds=True — newer stacks measurably degrade generation quality on identical weights.
• Text normalization: collapse whitespace and convert digits to Danish words (num2words, lang="da") before encoding.

Model details

Limitations

• Voice cloning transfers the broad character of a voice (timbre, pace, register), not a fine-grained identity — the speaker embedding is one vector per recording.
• Trained on a single speech domain; voices far outside it clone less faithfully.
• Sampling-based generation can occasionally mis-speak; for user-facing products, verify outputs with an ASR pass and resample on mismatch (a reference implementation ships with the plapre library's demo server).