cohere-transcribe-arabic-07-2026-dialectal

cohere-transcribe-arabic-07-2026-dialectal

Full fine-tune of CohereLabs/cohere-transcribe-arabic-07-2026 (2.07B, cohere_asr Conformer encoder-decoder) for multi-dialect Arabic speech recognition (undiacritized output).

Private / internal model. Evaluate on your own data before production use.

Results (932-clip held-out test set)

~22% relative WER cut over the (already strong) Arabic-specialized base. This is the model the base should be fine-tuned into: an earlier LoRA attempt on the same data made it worse (0.457 → 0.510, overfit). A full fine-tune with a low LR and best-checkpoint selection was the fix.

⚠️ This model overfits easily. Eval loss bottomed at step 1000 (0.342) and rose afterward, while WER stayed flat — the saved weights are that best checkpoint (load_best_model_at_end). Watch WER, not just loss, if you train further.

Model comparison — all Arabic ASR models

Same 932-clip held-out test set, same clean_text scoring (strip tashkil + tags, keep punctuation + dialect spelling) — so every row is directly comparable.

• Best fine-tuned: whisper-large-v3-turbo (WER 0.344), with cohere-transcribe-arabic a close second (0.357).
• Best zero-shot: cohere-transcribe-arabic (0.457, Arabic-specialized). A full fine-tune (all ~2B params, low LR) now improves it to 0.357; an earlier 32 GB LoRA attempt had instead degraded it (0.510, overfit) — full-parameter tuning with best-checkpoint selection was the fix.
• Streaming / low-latency: nemotron-3.5-asr.
• Parakeet-TDT-0.6b-v3 was tried but abandoned (European-only pretraining; cross-lingual transfer to Arabic converged far too slowly, WER ~0.93 after 3k steps).

Dataset

Fine-tuned on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private).

• ~40,000 train / 1,000 test clips, 16 kHz mono, single-channel.
• Multi-dialect Arabic: Levantine (Lebanese/Syrian), Maghrebi (Moroccan/Algerian/Tunisian), Egyptian, Gulf/Saudi, Sudanese, Iraqi, and MSA.
• Augmented: each clean clip is expanded with variants (noise, music, speed, voice, reverb/codec) via an augmentation column. Derived from oddadmix/dialectal-arabic-lahgtna-v2-smaller.

Preprocessing (applied to targets)

• Stripped tashkil/harakat (diacritics) and tatweel; removed non-verbal tags ([laughter], [exhale], [inhale], [mumble], [cough], timestamps, …).
• Kept dialectal consonants (گ ڨ چ پ ژ) — they encode real phonemes — and sentence punctuation. Output is undiacritized.
• Filtered: dropped clips > 30 s and a few corrupt rows (huge-text blobs). ~62% of rows had harakat, ~28% had non-verbal tags before cleaning. → 36,769 train / 932 test after filtering.

⚠️ Leakage note: the set is augmentation-expanded from shared source clips, so train/test may share source audio → held-out numbers can be optimistic. Use a leakage-free split for true numbers.

Training

• Base: CohereLabs/cohere-transcribe-arabic-07-2026 · full fine-tune (no LoRA)
• Effective batch 32 (per_device 2 × grad_accum 16) · LR 5e-6 (kept low — strong specialist) · 2 epochs (2300 steps) · best @ step 1000 (eval_loss 0.342)
• Precision: fp32 params + bf16 autocast + gradient checkpointing
• Hardware: a single 48 GB RTX A6000 (~43 GB peak, ~8.3 s/step)
• Metric: WER/CER on cleaned references (clean-text normalized), computed by per-sample generation (batched generation garbles this model's output)

Fine-tuning (reproduce)

The exact fine-tuning code is bundled in this repo (train_cohere_full.py, normalize.py, eval_hf_asr.py, ds_zero2.json, setup.sh) plus requirements.txt. See FINETUNE.md for the full walkthrough + lessons learned. Trained on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private) — swap in any HF audio dataset with audio + text columns. normalize.py is the shared text cleaning (strip tashkil + non-verbal tags, keep dialectal letters گ ڨ چ).

bash setup.sh                 # deps (transformers from source) + HF login

python train_cohere_full.py --output_dir cohere-ar-full \
  --per_device_train_batch_size 2 --gradient_accumulation_steps 16 \
  --per_device_eval_batch_size 2 --learning_rate 5e-6

python eval_hf_asr.py --model_type cohere --model cohere-ar-full   # WER / CER

For less memory / multi-GPU, accelerate launch train_cohere_full.py --deepspeed ds_zero2.json ....

Usage

# needs transformers from source (ships the cohere_asr architecture)
import torch, torchaudio
from transformers import AutoProcessor, CohereAsrForConditionalGeneration

repo = "oddadmix/cohere-transcribe-arabic-07-2026-dialectal"
proc = AutoProcessor.from_pretrained(repo)
model = CohereAsrForConditionalGeneration.from_pretrained(
    repo, torch_dtype=torch.bfloat16).to("cuda").eval()

wav, sr = torchaudio.load("clip.wav")          # 16 kHz mono
inp = proc(wav.mean(0).numpy(), sampling_rate=16000,
           language="ar", return_tensors="pt").to(model.device)
inp["input_features"] = inp["input_features"].to(model.dtype)   # keep ids long
ids = model.generate(**inp, max_new_tokens=256)
print(proc.tokenizer.batch_decode(ids, skip_special_tokens=True)[0])

Decode one bare array at a time. Passing a list + padding=True triggers a batched/padded path that badly degrades this model's generation (WER 0.45 → 0.65+).

Learnings & notes

• A strong specialist can be hurt by fine-tuning. Cohere's Arabic model has the best zero-shot WER of everything tried (0.457). LoRA on this data overfit and made it worse (0.510); only a full fine-tune with a low LR (5e-6) and best-checkpoint selection turned it into a net win (0.357).
• Watch WER, not loss. eval_loss bottomed at step 1000 and rose while WER held — more steps would have overfit. load_best_model_at_end keeps the right weights.
• Data cleaning matters most: keeping dialectal letters + stripping tashkil/tags (rather than aggressive letter-folding) was the biggest lever for a multi-dialect model.
• The processor concatenates the language prompt ⊕ transcript for the model's causal-LM loss — handled in the collator, no action needed.

Limitations

• Trained on augmented, partly synthetic multi-dialect data; real-world dialect coverage varies (Maghrebi is the hardest).
• Output is undiacritized and lower-cased for Latin tokens.
• Offline model (≤30 s chunks); not streaming.
• Overfits easily on smaller/noisier data — use a low LR and keep the best checkpoint.