karume-irodori-v4-small

Irodori-TTS v4 Small — Karume

What is this

A Japanese text-to-speech distribution: the Irodori-TTS v4 Small rectified-flow DiT converted into the WebGPU inference runtime Karume's container format (a .krm part sequence whose first part carries the graph and model descriptors). Runs as-is in the browser and in Deno.

• Nine graphs make up the chain: a shared Japanese ModernBERT backbone, the text_proj and caption_proj condition projectors, the reference-latent speaker encoder, the duration predictor, the DiT split in two — dit_context (the conditioning keys / values, once per generation) and dit (one forward per Euler step, plus one per classifier-free-guidance branch) — and the DACVAE codec (Aratako/Semantic-DACVAE-Japanese-32dim) as codec_decoder / codec_encoder.
• Text in, waveform out. generate() returns f32 mono samples at the codec's sample rate, ready for encodeWav. The decoder is run in tiles so that it also fits GPUs with the default 128MiB storage-buffer limit; the tiling is bit-exact against a single-shot decode.
• Voice cloning is wired up both ways: a reference speaker is passed either as audio (mono f32 — what decodeWav returns), which codec_encoder turns into a DACVAE latent, or as such a latent directly. Reference audio must already be at this distribution's own 48000 Hz: there is no resampler, and a mismatch is refused rather than silently converted. Unlike the decoder, codec_encoder is not tiled, so a long reference can exceed the default 128MiB storage-buffer limit.
• Not readable by the upstream implementation (it's a different container with an embedded graph); the reader is a pipeline that implements irodori/1.
• Exporter used for the conversion: karume/0.14.0. The distribution manifest is karume.json (karume/5).

Base weights and attribution

Converted into the container format — the original checkpoint is not distributed here.

• Weights: Aratako/Irodori-TTS-v4-Small, licensed MIT (as of retrieval). The training / inference implementation it comes with (Aratako/Irodori-TTS) is MIT as well; Karume's runtime contains none of that code — it is an independent implementation that reads these weights from its own container format.
• Text backbone: fine-tuned from sbintuitions/modernbert-ja-310m, licensed MIT (as of retrieval). It is redistributed here in the container format as the backbone component, so that license travels with this repository too.
• Codec: Aratako/Semantic-DACVAE-Japanese-32dim, licensed MIT (as of retrieval). Upstream ships it as a separate repository; it is redistributed here in the container format as the codec_decoder / codec_encoder components so that text-to-audio runs from this repository alone. Its weights derive from facebook/dacvae-watermarked, licensed Apache-2.0 (as of retrieval), by way of Aratako/Semantic-DACVAE-Japanese; the Apache License 2.0 text is included in LICENSE.md for the codec components.
• Changes made here (also listed in NOTICE.md): conversion into the Karume container format and quantization of the weights — every component is stored as f32 / f16 / i8 series, and the DiT (dit / dit_context) adds an i4 series rounded with GPTQ calibration (the quant table below says which storage each quant selects; f32 is not quantized). The codec stores its weight-normalized convolutions as effective weights, keeps only the mean half of quantizer.in_proj in codec_encoder, and bypasses the watermarking branch in codec_decoder; constant sub-expressions are precomputed. No retraining, no fine-tuning.

Models

model selects one of these; omitted, it is v4-small. quant defaults to that model's own default quant.

Usage

import { decodeWav, encodeWav } from "jsr:@karume/models";
import { Pipeline } from "jsr:@karume/models/irodori";

await using pipeline = await Pipeline.fromPretrained({
  repo: "hdae/karume-irodori-v4-small",
  // Pin a commit for reproducible builds — without it you track `main`, and a future
  // repo update (renamed files, new manifest format) may break your app.
  // Copy the full hash from this repo's "Files and versions" tab:
  // revision: "<full commit sha>",
}, {
  // model: "v4-small", // default — available: v4-small
  // quant: "i8-a8", // default — available: f16 / f32 / i8 / i8+dit4 / i8-a8
});

const audio = await pipeline.generate({
  text: "こんにちは、これはテストです。",

  // Voice Design — describe the voice in Japanese prose:
  // caption: "落ち着いた女性の声で、ゆっくりと丁寧に話している。",

  // Voice cloning — condition on a reference speaker. The WAV must already be
  // 48000 Hz — any channel count is averaged to mono; there is no resampler,
  // so a sample-rate mismatch is refused:
  // speaker: { audio: decodeWav(await Deno.readFile("reference.wav")) },
  // ...or pass a DACVAE latent you saved earlier instead of the audio:
  // speaker: { latent: savedLatent },

  seed: 42, // same seed + same inputs → same audio
  // durationSeconds: 5, // override the predicted utterance length (seconds)
});

await Deno.writeFile(
  "out.wav",
  encodeWav(audio.data, audio.sampleRate),
);

generate() returns { data, sampleRate, frames, seed, forwards }, where data is f32 mono already trimmed to the predicted length. generateLatent() is the same run stopped one stage earlier: it returns { data, frames, latentDim, seed, forwards } with the patched DACVAE latent, for callers that want the embedding rather than audio. caption and speaker are both optional: without them the voice is picked by the model alone, and the guidance branches for the missing conditions are skipped. Weights are fetched once and cached (verified against karume.json's size / sha256).

Model: v4-small

Quants

If no quant is given, it runs as i8-a8 (this model's recommended default). Per-file size and sha256 live in karume.json — verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / bf16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own.

Shape

Derived from the checkpoint's own config, and checked against the exported graphs when this repository was assembled.

• text: up to 256 tokens (BOS included), width 512
• caption: up to 512 tokens (BOS included), width 512
• reference speaker: up to 750 patched rows (4 latent frames each = 120s of audio), width 768
• latent: up to 750 frames at 25 Hz (30s), width 32
• audio: 48000 Hz mono, 1920 samples per latent frame

Defaults

The sampler knobs are fixed by the manifest — generate() takes none of them.

• steps: 40 Euler steps (initScale 0.999)
• guidance: text 3.0 / speaker 5.0 / caption 3.0, applied for t in [0.5, 1.0]
• duration: clamped to [0.5, 30.0] seconds (the duration predictor decides within that, unless durationSeconds is passed)

seed is the one knob the manifest does not carry — it defaults to 0, and the same seed with the same request gives the same audio.