qwen3-asr-0.6b-GGUF

Qwen3-ASR 0.6B — GGUF (ggml-quantised)

GGUF / ggml conversions of Qwen/Qwen3-ASR-0.6B for use with the qwen3-asr-main CLI from CrispStrobe/CrispASR.

Qwen3-ASR 0.6B is Alibaba's speech-LLM ASR model:

• 30 languages + 22 Chinese dialects with automatic language detection
• 6.42 % avg WER on the HuggingFace Open ASR Leaderboard
• Apache-2.0 licence
• Speech-LLM architecture: Whisper-style audio encoder (2D-conv subsampler + 18-layer Transformer + projector head, 896 → 1024) feeds frames into a stock Qwen3 0.6B LLM (28 layers, GQA 16/8, head_dim=128, Q-norm/K-norm, SwiGLU, RoPE θ=1e6) via embedding splice at <|audio_pad|> placeholder positions in a ChatML prompt. The LLM autoregressively generates the transcript.

This is the first speech-LLM in the CrispASR family — every other model in the set uses a dedicated CTC / transducer / encoder-decoder. The Qwen3-ASR runtime ships with a persistent KV cache so per-token decode is O(1) in cache size, not O(N) full re-forwards.

Files

All quantisations produce the correct transcript on samples/jfk.wav:

And so, my fellow Americans, ask not what your country can do for you; ask what you can do for your country.

2026-07 rebake — audio tower now Q8_0 (fixes long-audio repetition loops)

The sub-Q8 quantisations were re-baked with the 18-layer audio encoder kept at Q8_0 (previously Q4_0/Q4_K like the LLM body; ~90 MB larger per file). Diff-harness analysis on a real 145 s clip (CrispASR #218) showed sub-8-bit encoder weights compound per-block drift (block 0 cos 0.9996 → block 17 cos 0.973 vs the bf16 reference) until greedy decode flips into repetition loops ("hey, hey, hey, …") or an empty "language none" answer on long audio. With the Q8_0 tower the encoder output is back at cos ≈ 0.9997 and the full 145 s clip transcribes cleanly in one pass — matching the bf16 reference — with no repetition post-processing needed. Q8_0 / F16 files were never affected.

About the imatrix variant

-q4_k-imatrix.gguf is a Q4_K build quantised with an importance matrix: per-column activation statistics collected by running real audio through the F16 model (CRISPASR_IMATRIX_OUT), so the quantiser spends precision on the columns the model actually uses (the same idea as llama.cpp's llama-imatrix, but computed from audio rather than a text corpus).

Calibrated on a CC0 Mozilla Common Voice English + German sample, published at cstr/crispasr-imatrix-calib. Measured with the tools/imatrix_ab.py A/B harness (prefill first-token-logit cosine vs the F16 gold, held-out clips):

So the imatrix recovers a good part of the quantisation's logit error at no size cost. Calibration language coverage matters — an English-only corpus made the imatrix worse; the EN+DE mix is what produced the gains above.

Long-form caveat: the calibration corpus is short utterances. On long single-pass audio (--chunk-seconds 0, several minutes in one prompt) the imatrix variants can still drift into repetition where the plain -q4_k.gguf stays clean — for long-form use prefer -q4_k.gguf or -q8_0.gguf.

The mel filterbank from WhisperFeatureExtractor is baked into the GGUF as audio.mel_filters (along with audio.mel_window), so the C++ runtime computes the log-mel spectrogram natively without needing torch / librosa / scipy at inference time.

Quick Start

# 1. Build the runtime
git clone https://github.com/CrispStrobe/CrispASR
cd CrispASR
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target qwen3-asr-main

# 2. Download a quantisation
huggingface-cli download cstr/qwen3-asr-0.6b-GGUF \
    qwen3-asr-0.6b-q4_k.gguf --local-dir .

# 3. Transcribe
./build/bin/qwen3-asr-main \
    -m qwen3-asr-0.6b-q4_k.gguf \
    -f your-audio.wav -t 8

Audio must be 16 kHz mono 16-bit PCM WAV. Pre-convert with:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

Performance

Measured on samples/jfk.wav (11 seconds), Apple-class 4-core CPU:

Q4_K runs slightly faster than realtime with no quality loss on this clip.

Architecture

Implementation notes (correctness)

The C++ runtime is verified to F16 numerical precision against the PyTorch reference at every architectural boundary on samples/jfk.wav:

Bugs that would have been hours of debugging

A few non-obvious gotchas the port had to handle:

1. ggml_permute semantics are inverted from the obvious reading: permute(t, p0, p1, p2, p3) means "source axis i goes to NEW position p_i", not "new axis i comes from source axis p_i".
2. PyTorch hooks fire pre-GELU when registered on an nn.Conv2d module — the F.gelu is applied externally in the forward function.
3. cu_seqlens is GPU-only: eager_attention_forward (used on CPU) ignores cu_seqlens and does standard full self-attention. The "windowed attention" path only kicks in for FlashAttention2 on GPU. Don't apply the windowed mask on CPU — the reference produces full-attention output.
4. WhisperFeatureExtractor.mel_filters shape is (n_freqs=201, n_mels=128), not (n_mels, n_freqs) as the parameter ordering might suggest.
5. Qwen3 attention output width is hd × n_q_heads = 2048, not d_model = 1024. The o_proj is (2048 → 1024), so the attention output is reshaped to (2048, T) before o_proj.
6. mrope sidestep: Qwen3-ASR uses interleaved multi-modal RoPE with mrope_section=[24,20,20]. For text-only or 1D-position input (which includes our spliced audio frames), the three mrope sections all receive identical position_ids and collapse to standard 1D RoPE. The simpler RoPE matches the reference perfectly for our use case.

See qwen3-asr-todo.md in the runtime repo for the complete work log.

How this was made

1. The HF safetensors model was converted to GGUF F16 by models/convert-qwen3-asr-to-gguf.py. All 612 tensors map cleanly. The mel filterbank (from WhisperFeatureExtractor.mel_filters) and Hann window are baked into the GGUF as audio.mel_filters / audio.mel_window.
2. Quantised variants are produced by crispasr-quantize (the same llama.cpp-style quantiser used for the other GGUF releases in this family).
3. Inference is implemented in src/qwen3_asr.{h,cpp}: the encoder and the LLM each run as one ggml graph, with a persistent F32 KV cache (head_dim, max_ctx, n_kv_heads, n_layers) shared between prefill and per-token decode steps.

Reference implementation

predict-woo/qwen3-asr.cpp (MIT) was read for architecture discovery and tensor name mapping. No source code was vendored — the CrispASR runtime is a re-implementation in this repo's existing FastConformer / cohere-style ggml infrastructure, sharing structures with the four other ASR runtimes in the family.

Supported languages

ar cs da de el en es fa fi fil fr hi hu id it ja ko mk ms nl pl pt ro ru sv th tr vi yue zh plus 22 Chinese dialects (auto-detected at inference time).

Attribution

• Original model: Qwen/Qwen3-ASR-0.6B (Apache-2.0). Alibaba Cloud Qwen team.
• GGUF conversion + ggml runtime: CrispStrobe/CrispASR — community contribution.
• Reference implementation: predict-woo/qwen3-asr.cpp (MIT) — used for architecture discovery only, no code vendored.

Related

• C++ runtime: CrispStrobe/CrispASR
• Sister releases in the same family:
  • cstr/cohere-transcribe-03-2026-GGUF — Cohere Transcribe 2B (Open ASR Leaderboard #1)
  • cstr/parakeet-tdt-0.6b-v3-GGUF — Parakeet TDT 600M (free word timestamps)
  • cstr/canary-1b-v2-GGUF — Canary 978M (speech translation)
  • cstr/canary-ctc-aligner-GGUF — universal multilingual forced aligner

License

Apache-2.0, inherited from the base model.

Provenance and EU AI Act Art. 53 note

• Upstream model: Qwen/Qwen3-ASR-0.6B — published by Qwen.
• Upstream licence: apache-2.0. This repository redistributes under the same terms; it grants no rights the upstream licence does not.
• What was done here: format conversion and/or quantisation only (GGUF/GGML). No training, no fine-tuning, no merging, no distillation, no change to architecture, vocabulary or capability. Only the numeric representation of the upstream weights differs.
• Training data: documented — where it is documented at all — by the upstream provider; see the upstream model card. No training data was used, added or selected by this repository.
• Provider status: under Regulation (EU) 2024/1689 the upstream authors remain the provider of this model. Converting the serialisation format does not make this repository the provider of a new general-purpose AI model, and no such claim is made. Questions about training content, copyright policy or model capability belong upstream.