tr-hi-s2st-v0.1

Version: v0.1.0 — step-15000 checkpoint (first public release; superseded by v0.3). Versions are git tags in this repo; load a specific one with revision="v0.1.0". See Version history at the bottom.

🔜 A newer version exists: tr-hi-s2st-v0.2 fixes the text stream (the Phase-0 padding-loss fix) so the inner-monologue actually learns. If you want the later recipe, use v0.2.

📒 Data clarification

This checkpoint was trained on the project's synthetic corpus — tiny-aya-translate/tr-hi-mimi-encoded (FLORES + OPUS-100 + machine-translated conversational text, rendered with multi-voice TTS; 1,178,302 train / 62,036 val pairs, verified from the training log). Earlier wording in this card called the data "FLEURS" — that was a mislabel: FLORES (parallel text) is not FLEURS (read-speech audio), and no FLEURS audio is in this set. (Note: the later v0.2 run accidentally trained on a FLEURS sibling dataset; v0.1 here used the intended synthetic data.)

TinyAya Stage 2 — Turkish ↔ Hindi Speech-to-Speech Translation (LoRA)

Stage-2 simultaneous-translation adapter for Turkish↔Hindi speech-to-speech translation, trained on TPU v6e-8. This repo ships the authors' trained deltas only — a LoRA adapter over CohereLabs/tiny-aya-base plus the custom projection / Moshi depth-decoder / audio-head / embedding tensors.

• Developed by: tiny-aya-translate
• Funded by: Google TPU Research Cloud (TRC) — see Acknowledgements
• Model type: parallel two-stream S2ST (Cohere2 + LoRA + custom heads; frozen Moshi depth decoder)
• Languages: Turkish (tr), Hindi (hi)
• License: CC-BY-NC-4.0 — non-commercial, inherited from the base model (the training/eval code is Apache-2.0; see scope below)
• Finetuned from: CohereLabs/tiny-aya-base (+ Moshi / Mimi from Kyutai)
• Newer version: tiny-aya-translate/tr-hi-s2st-v0.2

⚠️ License scope — read first

The weights in this repo (LoRA adapter + custom heads) are CC-BY-NC-4.0: they are derivatives of CohereLabs/tiny-aya-base, which is CC-BY-NC-4.0, so the non-commercial term is inherited and cannot be relicensed away. The authors' separate training/eval code is Apache-2.0. Component terms (mirrored from THIRD_PARTY_NOTICES.md):

• CohereLabs/tiny-aya-base base weights → CC-BY-NC-4.0. Not included here; obtain it from Cohere and comply with its terms.
• Moshi / Mimi (depth decoder + audio codec) → CC-BY-4.0 (attribution; Kyutai's Moshi code is separately Apache-2.0).
• Source data (FLORES, OPUS-100, conversational MT) and the TTS-generated audio → see the dataset card for per-source terms (e.g. FLORES is CC BY-SA 4.0; OPUS-100 sub-corpora and the TTS-model outputs carry their own licenses). Verify before redistribution.

What this is

Training procedure

• Init: LoRA (r=16) on the tiny-aya-base backbone; Moshi depth decoder initialized from kyutai/moshiko; projection / per-codebook audio heads / audio & text embeddings trained from scratch.
• Recipe: effective batch 256 (b=8 × grad-accum 4 × 8 chips), bf16, FSDPv2 SPMD, cosine LR (lr_lora 1.5e-4), 500 warmup steps, max_frames=300, 8 codebooks, hierarchical codebook loss (text_weight 0.1, audio_weight 1.0).
• Compute: single-host Cloud TPU v6e-8 (spot), ~22 h, 15,000 steps (≈ 3.3 epochs over 1,178,302 train pairs).

Convergence note (honest): the audio loss plateaued by ~step 8,000 (≈1.7 epochs) and did not improve over the final 7k steps. The text / inner-monologue stream did not learn in this run (loss ≈ random), pending a data-pipeline fix. So this checkpoint reflects the audio-translation capability of the recipe at convergence, not an under-trained model. This is fixed in v0.2 (the text-padding-weighted loss), where the text stream learns.

Evaluation

Speech-translation quality is measured with ASR-BLEU (Whisper transcribes the generated target audio, BLEU vs. reference target text) and DNSMOS (naturalness). ASR-BLEU is implemented in scripts/eval_checkpoint.py; DNSMOS is not yet wired up. Pending for this release (v0.1.0); will be filled in the YAML model-index + below.

Intended use & limitations

• Intended: research on low-resource speech-to-speech translation and simultaneous translation; a Stage-2 checkpoint, not a production system.
• Limitations: trained on synthetic multi-voice TTS audio (FLORES / OPUS-100 / conversational text) — expect degradation on real, spontaneous, or noisy audio and on voices outside the TTS set; two language directions only; the text stream did not learn in this version (root cause found 2026-07-08: a loader path bug — the corpus's word-level alignments ship under {stem}.{src,tgt}.alignments.json at the dataset root, which the v0.1 loader never resolved, so text supervision was silently all-padding; fixed in v0.3); generation is autoregressive and not optimized for latency here.
• Bias & risks: a fixed set of TTS voices — fairness across real speakers, dialects, accents, code-switching, and spontaneous speech is untested. Speech translation can mistranslate, omit, or fabricate content; not for high-stakes use.

Inference quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "CohereLabs/tiny-aya-base"  # obtain per Cohere license
ADAPTER = "tiny-aya-translate/tr-hi-s2st-v0.1"  # this repo

tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, trust_remote_code=True)
model = PeftModel.from_pretrained(base, ADAPTER)  # loads adapter_model.safetensors

# Then attach the custom heads (projection / depth_decoder / audio_heads /
# text_embed / model_audio_embed *.pt) and the Mimi codec; see the training
# repo for the full composite + generation loop.

The full speech→speech pipeline (Mimi encode → backbone+depth-decoder → Mimi decode) lives in the training repo (src/model/composite.py).

Links

• Training run (W&B): https://wandb.ai/cataluna84/tinyaya-stage2-tpu/runs/b7fr72u5 — full config, loss curves, system/throughput metrics, and the model artifact (tinyaya-stage2-tr-hi-v6e-v2:v0).
• Checkpoints: the training-time GCS bucket was decommissioned after the v0.3 release; published weights live on the Hub.
• Dataset: https://huggingface.co/datasets/tiny-aya-translate/tr-hi-mimi-encoded
• Code: https://github.com/tiny-aya-simultaneous-translation/model
• Next version: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.2

Version history

Versions are git tags in this repo (HF convention: one checkpoint per repo, versioned with tags). Load a specific one with revision=:

PeftModel.from_pretrained(base, "tiny-aya-translate/tr-hi-s2st-v0.1", revision="v0.1.0")

When a materially different model is trained (new data / architecture), it goes in a new repo — here, tr-hi-s2st-v0.2 (the new_version: field shows a forward banner on the Hub).

Acknowledgements

This model was trained on Cloud TPU v6e-8 hardware generously provided by Google's TPU Research Cloud (TRC) program. We thank the TRC team for supporting this research. See NOTICE and THIRD_PARTY_NOTICES.md for component licenses.

Citation

@misc{tinyaya_tr_hi_s2st_v0_1,
  title  = {TinyAya: Turkish-Hindi Speech-to-Speech Translation (v0.1)},
  author = {tiny-aya-translate},
  year   = {2026},
  note   = {Cohere2 + frozen Moshi depth decoder, LoRA, trained on Google TRC TPU v6e-8},
  url    = {https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.1}
}

Where this sits

The v0.3 speech-to-speech pipeline, end to end:

tr-hi-parallel-text          text triples (en pivot -> tr / hi)
        |  TTS
tr-hi-parallel-speech-v2     synthetic speech + QC signals
        |  Mimi encode
tr-hi-mimi-encoded           8-codebook tokens + word alignments
        |  Stage-2 training
tr-hi-s2st-v0.3              the released model

Code

Project

TinyAya Stage 2 — Turkish⇄Hindi speech-to-speech translation with a text inner-monologue: a LoRA-adapted Cohere2 backbone driving a frozen Moshi depth decoder over Mimi codes.

The v0.3 run covered 76,250 steps / 2.07 epochs on a Cloud TPU v6e-16 (best val composite 2.8199 @ step 76,000). Read honestly: the text inner-monologue learns to translate (free-run chrF++ ~25.7 / 25.1), while intelligible audio synthesis remains the frontier (ASR-chrF++ 3.7 / 9.6 against a 92.1 / 86.6 ground-truth-audio ceiling) — bounded by the frozen depth decoder, not by translation understanding.

• Results: v0.3 evaluation report
• Training run: W&B xzcb60bl · emergence report
• Blog: Adapting Moshi for Low-Resource Speech Translation

Compute for the v0.3 run was provided by Google's TPU Research Cloud (TRC).