Sophea-Canary-ASR

Sophea-Canary-ASR

A bilingual (Greek + English), multitask speech model fine-tuned from nvidia/canary-1b-v2 (1 B, offline attention-encoder-decoder, FastConformer).

Base model vs. this model

How much did fine-tuning move each task? Both the base nvidia/canary-1b-v2 and this model were scored on the same held-out test manifests with the same validation-path harness and literal WER (casing + punctuation), so these deltas are directly comparable.

Reading it: Greek fine-tuning is the whole story — FLEURS WER drops by more than half (30.9 -> 12.7) and Common Voice collapses from 37.3 to 2.9. English ASR and Greek->English translation were never trained here; they only moved by ~1 point, i.e. they were retained rather than forgotten. Canary's source_lang/target_lang prompt-conditioning is what keeps the English and translation pathways intact while the Greek pathway is rebuilt.

What it does

One model does four jobs:

It also handles medical and legal Greek domains (in the fine-tuning mix). This is the offline, high-accuracy model of a two-model Greek ASR system; a separate streaming 0.6 B model serves the real-time path.

Highlights

• Halves Greek WER vs the streaming baseline (25.5 % -> 12.7 % FLEURS) and drives Common Voice to 2.9 %.
• Bilingual for free. Despite fine-tuning on ~360 h of mostly-Greek data, English ASR held at 12.9 % (base canary-1b-v2: 12.0 %) — Canary's source_lang/target_lang prompt-conditioning preserves the English pathway (no catastrophic forgetting).
• Translation for free. Greek->English speech translation (BLEU 24.1) was retained from the base through ASR-only fine-tuning — the prompt selects the task, so the AST pathway survives.

Usage

Requires NeMo (pip install nemo_toolkit[asr]).

from nemo.collections.asr.models import EncDecMultiTaskModel
from huggingface_hub import hf_hub_download

ckpt = hf_hub_download("KIEFERSA/Sophea-Canary-ASR",
                       "canary1bv2_el_stage3.nemo")
model = EncDecMultiTaskModel.restore_from(ckpt)

# Greek ASR
print(model.transcribe(["greek_audio.wav"], source_lang="el", target_lang="el", pnc="yes"))

# English ASR
print(model.transcribe(["english_audio.wav"], source_lang="en", target_lang="en", pnc="yes"))

# Greek -> English speech translation
print(model.transcribe(["greek_audio.wav"], source_lang="el", target_lang="en", pnc="yes"))

Prompt-sensitivity note. This model is fine-tuned with the canary2 prompt and pnc=yes. Pass source_lang/target_lang explicitly (as above) and keep pnc=yes for results matching the reported metrics. Audio should be 16 kHz mono.

Training

• Base: nvidia/canary-1b-v2 (1 B, offline AED, FastConformer encoder).
• Method: full fine-tune via a custom NeMo launcher that restores the pretrained model and swaps the dataset (avoiding the tokenizer rebuild ..), with a 3-stage warm-start chain (each stage initialized from the previous endpoint):
  1. stage-1 — ~250 h Greek ASR (FLEURS + Common Voice + YODAS + TEDx + targeted TTS).
  2. stage-2 — + ~80 h clean Greek (medical, legal, parliamentary) -> 330 h.
  3. stage-3 (this model) — + 30 h English ASR (LibriSpeech-clean) -> 360 h.
• Optimizer: AdamW, cosine schedule, lr ~8e-6 (stage-3), bf16, Lhotse dynamic batching.
• Checkpoint selection: best Greek val_wer (0.0577 at stage-3).

Data

Limitations

• Offline only. Full-context AED — not for streaming/real-time use.
• Translation ceiling. Greek->English BLEU (~24) is inherited from the base; an explicit AST fine-tune on machine-translated data did not improve it (you cannot out-train your labels). Real parallel data would be needed to push past this.
• Domain bias. Strongest on read speech (FLEURS) and Common Voice; other domains (e.g. heavy dialect, far-field, overlapping speech) are untested.
• Common Voice over-specialization. The validation mix is CV-heavy, so CV WER (2.9 %) is partly in-distribution; FLEURS (12.7 %) is the more conservative real-world estimate.

License

Released under CC-BY-4.0, inheriting the license of the base model nvidia/canary-1b-v2.

Citation

If you use this model, please credit this repository and the base model:

@misc{kiefer2026canarygreek,
  title  = {Sophea-Canary-ASR: a bilingual Greek+English multitask speech model},
  author = {Kirouane, Ayoub},
  year   = {2026},
  howpublished = {Hugging Face, KIEFERSA},
  note   = {Fine-tuned from nvidia/canary-1b-v2}
}