stt_bn_qwen3_asr

stt_bn_qwen3_asr

Research release — not recommended for production or latency-sensitive use. This adapter does not establish an accuracy advantage over hishab/titu_stt_bn_conformer_large, which is more accurate and roughly 19× faster in the batch-1 timings below, and it is statistically tied with SayedShaun/bengali-whisper-medium, which is about 3× faster. It requires the separate Qwen3-ASR-1.7B base model and a hand-written unsupported-language prefill; the normal transcription API rejects Bengali. Evaluation is FLEURS read speech only — no noisy, conversational, long-form, dialectal, safety or subgroup validation is claimed. Gating records downloader account details; it is not vetting or endorsement.

A Bengali adapter for Qwen3-ASR-1.7B, an LLM-decoder ASR model whose supported-language list does not include Bengali. Zero-shot, the base model transcribes Bengali speech into Devanagari at 77.12% WER. Training 38.01M parameters — 1.87% of the 2.04B-parameter system — on 984 hours of fully human-supervised Bengali brings it to 16.54%.

This is a LoRA adapter, not a full model. It loads on top of the base checkpoint.

Results

FLEURS Bengali test split (920 utterances), greedy, batch 1, no language model. Every row is our own measurement on one RTX A5000, one model resident at a time, through an identical audio path.

Compared pairwise on identical utterances, clustered by the 349 distinct reference sentences:

• Tied with Whisper Medium: +0.31 pp, 95% CI [−0.56, +1.18] — contains zero. Whisper Medium is also ~3× faster per clip, so it is the better practical choice on this benchmark. What this model has over it is a lower CER and an LLM decoder.
• Behind Conformer Large: +2.16 pp, 95% CI [+1.14, +3.17] — excludes zero, so the gap is real. On CER the two are level: +0.20 pp, 95% CI [−0.37, +0.74].

This model is not the most accurate or the fastest Bengali ASR in this table. An off-the-shelf CTC Conformer is both more accurate and ~19× faster. What this demonstrates is that a language absent from an LLM-decoder ASR model can be added for 38M trained parameters.

Two scoring details that change these numbers

Unicode normalization is applied (NFC). Bengali writes য় as either U+09DF or U+09AF + U+09BC (likewise ড় and ঢ়) — canonically equivalent, different bytes. The references and this model use the decomposed spelling; Whisper and wav2vec2 emit the precomposed one. Comparing raw strings counts every occurrence as a substitution and inflates Whisper Medium from 16.23% to 27.88%. Scoring without NFC would make this model look 11 points better than it is.

FLEURS validation is excluded. The training corpus includes FLEURS train, and FLEURS reuses sentences across splits, so 88 of the 150 distinct sentences in the validation split appear verbatim in training data. Scoring on test+validation would report 16.03% instead of 16.54%, flattering this model and not the third-party checkpoints. The test split has zero overlap with training.

Usage

Bengali is not in the base model's language list, so apply_transcription_request() rejects it. The assistant turn must be prefilled by hand with language Bengali<asr_text> and that prefill stripped from the decode. Get this wrong and the model transcribes into Devanagari without complaining.

# pip install "transformers==5.16.1" peft soundfile
import torch, soundfile as sf
from peft import PeftModel
from transformers import AutoProcessor, Qwen3ASRForConditionalGeneration

BASE = "Qwen/Qwen3-ASR-1.7B-hf"
proc = AutoProcessor.from_pretrained(BASE)
model = PeftModel.from_pretrained(
    Qwen3ASRForConditionalGeneration.from_pretrained(
        BASE, dtype=torch.bfloat16, attn_implementation="sdpa"),
    "ehzawad/stt_bn_qwen3_asr").merge_and_unload().to("cuda").eval()

# Worth 6.3x together, and free. See "Decode speed".
model.generation_config.cache_implementation = "static"

wav, sr = sf.read("audio_16k_mono.wav", dtype="float32")   # 16 kHz mono
conv = [{"role": "user", "content": [{"type": "audio", "audio": wav}]},
        {"role": "assistant",
         "content": [{"type": "text", "text": "language Bengali<asr_text>"}]}]
inputs = proc.apply_chat_template([conv], tokenize=True, return_dict=True,
                                  continue_final_message=True)
inputs = {k: (v.to("cuda", dtype=torch.bfloat16) if hasattr(v, "to")
              and v.is_floating_point() else v.to("cuda") if hasattr(v, "to") else v)
          for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=512, do_sample=False, use_cache=True)
text = proc.batch_decode(out[:, inputs["input_ids"].shape[1]:],
                         skip_special_tokens=True)[0]
print(text.split("<asr_text>")[-1].strip())

Decode speed

Autoregressive decoding is the cost here, and the default configuration is far from the best one. Measured on an RTX A5000 over 1,322 utterances:

6.34×, with WER moving +0.051 pp on a 95% CI of [−0.035, +0.141] — an interval containing zero, verified by rescoring rather than assumed.

The accuracy table above is measured on the default runtime, as are all the other models, so the comparison stays like-for-like. Scored on the recommended merged + static-cache path instead, this adapter measures 16.65% WER / 4.71% CER on the same 920 test utterances — 0.11 pp worse, from bf16 reduction-order changes at near-ties. That is the number you should expect from the usage snippet above.

99.7% of a clip is the decode loop; the audio encoder is 0.2%. Profiling found 1,602 kernel launches per generated token with the GPU idle 58% of the time — RMSNorm running as four separate kernels and the dynamic cache reallocating K and V every layer, every step. Merging removes 196 separately-dispatched LoRA modules; the static cache removes the reallocation.

Even so, this remains ~19× slower per clip than a CTC Conformer (1,461 ms vs 78 ms), which emits an utterance in one forward pass. Two structural reasons: 2.04B parameters, and Qwen's vocabulary has no Bengali merges, so Bengali costs ~6.4 byte-fallback tokens per word (~130 tokens per clip).

What was trained

The encoder really is frozen. The audio tower uses out_proj/fc1/fc2 while the decoder uses o_proj/gate_proj/up_proj/down_proj, so those four cannot match the encoder by name; q_proj/k_proj/v_proj exist in both and are therefore pinned to fully-qualified language_model.layers.N.* names.

Fine-tuning used a warmup–stable–decay schedule over one epoch at effective batch 32, with a fused linear cross-entropy that never materialises the [B, T, 151936] logit tensor.

Training data

984 hours, all human-read or human-transcribed with human verification. No pseudo-labels, no alignment-mined text, no synthetic audio.

Licence caveat, stated plainly. OpenSLR-53 is CC-BY-SA-4.0. ShareAlike requires derivatives to carry the same or a compatible licence, and CC-BY-NC-SA adds a NonCommercial restriction, so if trained weights are held to be adapted material of their training data, this adapter's licence is not compatible with that source. Whether model weights are legally derivative of training examples is unsettled and jurisdiction-dependent. This is disclosed rather than resolved: review it before any commercial or redistribution decision. Kathbath's card is also internally inconsistent (metadata says CC-BY-4.0; its detailed section says only the packaging is CC0), and SUBAK.KO includes material sourced from broadcast and YouTube.

The base model is Apache-2.0 © Alibaba; that licence governs the base-derived portions.

Failure modes and latency tail

Means hide the behaviours that decide whether a model is usable. Over all 920 test utterances:

Zero degenerate outputs. The script result is the one worth noting: the un-adapted base model transcribes Bengali into Devanagari, and after training that behaviour is entirely gone.

Latency distribution, default runtime, batch 1 (ms/clip):

The tail matters more than the mean for an autoregressive model: cost scales with how much text is emitted, so long utterances are disproportionately slow. The merged + static-cache path divides these by roughly 6.3.

Limitations

• FLEURS is short, clean read speech. It says nothing about conversational, noisy, dialectal, or long-form audio.
• Batch 1, greedy, single run per model; speed figures carry no interval.
• ~19× slower per clip than a CTC model and tied on accuracy with a model ~3× faster. Choose it for the LLM decoder, not for the leaderboard position.
• 25-second segmentation is applied to longer audio; the base model's window is 30 seconds.

Reproducing the numbers

Everything needed to recompute the table, stated so a disagreement can be located rather than guessed at:

• Scoring surface: punctuation mapped to word boundaries (["“”‘’'।,.!?;:()\[\]{}—–-] → space, whitespace collapsed), then Unicode NFC. WER and CER via jiwer on whitespace-split words / raw characters.
• Decoding: greedy, num_beams=1, max_new_tokens=512, batch 1, no external language model, no context biasing.
• Audio: 16 kHz mono, 25-second segmentation for longer clips.
• Intervals: percentile bootstrap, 10,000 resamples, seed 20260903, resampling the 349 distinct reference sentences rather than the 920 recordings — several speakers read the same sentence, so recordings are not independent.
• Base: Qwen/Qwen3-ASR-1.7B-hf at revision bcd2b5b7f32b480ab5790554cfa8347f246a14f3, transformers==5.16.1, peft==0.20.0, bfloat16.
• Comparators are pinned to commit hashes and were run by us, not quoted from their authors.

Sentence-clustering handles repeated prompts. It does not handle repeated speakers: the FLEURS manifest used here carries no speaker identifiers, so speaker overlap between splits cannot be excluded by identifier and the intervals should be read with that caveat.

A note on gating

Access is auto-approved: any logged-in Hugging Face user is granted access immediately after sharing their username and email. That is a record of who downloaded it, not a review — it does not vet users, enforce the licence, prevent redistribution, or imply the model is fit for a given use. If you would rather not share those details, the benchmark code and the full evaluation protocol are available separately and do not require access to these weights.

Citation

@misc{zawad2026bnqwen3asr,
  author = {Emrul Hasan Zawad},
  title  = {stt_bn_qwen3_asr: a Bengali adapter for Qwen3-ASR-1.7B},
  year   = {2026},
  url    = {https://huggingface.co/ehzawad/stt_bn_qwen3_asr}
}