vietnamese-lalm-asr-lora

Vietnamese LALM ASR — LoRA adapter (r16)

A Large Audio Language Model (LALM) for Vietnamese speech-to-text, built the cheapest way possible: bolt a frozen audio encoder onto a small frozen LLM through a thin trainable projector, then unlock the LLM a little at a time with LoRA.

This repository releases the LoRA + projector adapter (~31 MB) plus a detailed write-up of the training journey, the problems hit, and the measured results. It is primarily a documentation / archival release — see Intended use & limitations below.

Tóm tắt (VI): Model ghép audio encoder (FastConformer, đóng băng) + LLM nhỏ (Sailor 0.5B, đóng băng) qua một projector tuyến tính, rồi mở khóa LLM bằng LoRA. Repo này chia sẻ **adapter

• kinh nghiệm + kết quả** của quá trình fine-tune ba nấc. Chi tiết số liệu ở phần Results.

---

Model summary

The only trainable weights are the projector and the LoRA matrices. The encoder and the LLM (including its embedding and lm_head) stay frozen — the LLM's 151936-token vocabulary already covers both Vietnamese and English, which is exactly what a closed-vocabulary CTC/transducer decoder cannot do.

---

Intended use & limitations

• What this repo is: the trained adapter (adapter.pt) + a full engineering write-up. It is meant for people studying how to stand up an X-LM (audio/vision/video encoder + LLM) on a small compute budget, and for archiving the exact weights behind the numbers below.
• ⚠️ Not runnable stand-alone. The adapter contains only the LoRA deltas and the projector. To run inference you also need the frozen SALM base: the in-house FastConformer encoder and the Sailor-0.5B-Chat LLM, wired together with NeMo speechlm2. The frozen encoder checkpoint is not released here, so this release does not reproduce end-to-end inference on its own.
• Domain: tuned for Vietnamese, including conversational/telephony-style speech; English technical terms are handled via the LLM's open vocabulary.
• Not for: safety-critical transcription, languages other than Vietnamese, or as a general chat model.

---

Architecture & data flow

Audio 16 kHz mono
   → Preprocessor (mel-80, 25 ms window / 10 ms hop)
   → FastConformer encoder (frozen)  — subsampling ×8, 1 frame ≈ 80 ms, 512-dim
   → Projector 512 → 1024 (trainable)
   → Sailor LLM (frozen + LoRA)  — audio vectors spliced into a fixed text prompt
   → Transcript tokens

Input is audio plus a fixed text prompt containing an audio-locator tag; the projected audio vectors replace that tag inside the prompt, so the task can be changed by editing the prompt rather than the architecture. A 10-second utterance becomes ~126 audio frames fed to the LLM.

---

Training recipe — three-stage curriculum

The guiding principle: unlock parameters in order of return on compute. Align the projector before letting gradients touch the LLM; open the LLM with ~1% LoRA before considering anything deeper.

1. Stage 1 — projector only. Freeze encoder + LLM, align the 0.53M-param projector. Proves the pipeline runs and gives the next stage a warm projector. Ceiling is low (WER 27–71%) because a frozen chat LLM prefers to paraphrase rather than transcribe.
2. Stage 2 — unlock the LLM with LoRA. Insert LoRA (r=16, α=32, dropout 0.05) into all 7 projections per layer (q/k/v/o_proj, gate/up/down_proj). WER drops 2–4× on every set; English code-switch recall jumps thanks to the LLM's open vocab. Two weaknesses surface: spoken-digit recall regresses, and the model ignores context-biasing prompts.
3. Stage 3 — retrain on data that patches the weak spots. Warm-start from the stage-2 adapter; add synthetic spoken-digit data, widen the broad-domain sets, and interleave prompts carrying a term list to teach the context-biasing channel. WER improves on all 12 evaluation sets and the biasing channel starts working.

Optimizer: AdamW, LoRA lr 1e-4, projector lr 2e-4, weight decay 1e-3, cosine schedule with warmup, batch 16, warm-started between rounds. Because the streaming (lhotse iterable) data has no declared length, all control is by step, not epoch — stop by max_steps, validate by integer val_check_interval, and checkpoint the adapter every N steps.

This adapter is the v3 round checkpoint (a full run of the stage-3 curriculum with the enlarged data mix).

---

Results

Word Error Rate (%, lower is better) on public Vietnamese benchmarks. LALM and external baselines are measured on the first 300 utterances per set; the in-house transducer (s7) is measured on the full suite, so gaps under ~1 point should not be read as real differences.

Specialized recall (%, higher is better) — English code-switch terms and spoken digits:

Reading the numbers:

• With ~1.1% trainable parameters, the LALM matches and on 4 sets beats the same-encoder transducer (s7), and reaches strong WER on the in-house conversational set (vss 11.97).
• Open vocabulary is the real differentiator. Code-switch recall (52.10) is nearly double the closed-vocab transducer (26.74) — chunkformer wins broad-domain WER but its CTC decoder is stuck at 11.38 on code-switch. This capability cannot be reached by fine-tuning a closed-vocab model further; it needs the LLM decoder.
• Broad-domain Vietnamese still has clear public champions (chunkformer leads 8/12 sets) — the gap is data and training scale, not architecture.
• The context-biasing channel works when trained: feeding the correct term list at inference lifts unseen-term recall from 15.38 to 38.46 (oracle upper bound).

---

Key lessons (the "experience" part)

• The projector is only a coupling. To change the LLM's behaviour you must let gradients reach it — a 0.5M-param linear layer alone cannot teach a chat LLM to transcribe verbatim.
• Changing the decoder changes the error distribution. Re-measure every specialized skill (e.g. digit recall) after swapping architecture; the aggregate WER hides regressions.
• Measure the oracle before training a control channel. If injecting the answer into the prompt moves the metric, the channel is worth training; if the oracle does not move, stop early.
• Patch weak skills with targeted synthetic data — far cheaper than collecting more real audio. Synthetic digit data halved WER around digit strings, though lifting exact per-digit recall past a strong closed-vocab baseline needs more/diverse data.
• Control training by step, not epoch, for any length-less streaming data pipeline.
• Checkpoint the trained params periodically — the cheapest insurance against infra failures; here it is a 31 MB file, and it let a hung run resume by warm-starting from the last checkpoint.
• Standardize a results registry (one JSON schema) from day one so in-house and external models live in one comparison table; a new measurement campaign is just adding a row.
• The whole recipe is modality-agnostic — swap the audio encoder for a vision/video encoder and the three-block + curriculum recipe stays the same.

---

Files in this repo

• adapter.pt — the trained adapter (~31 MB): projector weights + LoRA matrices for the frozen Sailor-0.5B-Chat LLM. Load into a NeMo speechlm2 SALM whose encoder is the frozen FastConformer and whose LLM is sail/Sailor-0.5B-Chat.
• README.md — this card.

---

Acknowledgements & license

• Decoder LLM: Sailor-0.5B-Chat (Apache-2.0).
• Framework: NVIDIA NeMo speechlm2.
• Audio encoder derived from the NeMo FastConformer family, fine-tuned in-house on Vietnamese.
• Public evaluation benchmarks: VIVOS, VietBud500, Common Voice, FLEURS, FOSD, LSVSC, VLSP2020, VietMed, VietSuperSpeech.

Released under Apache-2.0 (matching the Sailor base and NeMo framework). The adapter weights are the authors' own contribution.