qwen3-asr-0.6b-int4-v080

Qwen3-ASR-0.6B int4-AWQ — v0.8.0 Deploy Bundle

TensorRT-Edge-LLM v0.8.0 ASR engine bundle for NVIDIA Jetson (Orin-class). Mirrors the layout of qwen3-tts-base-v080-deploy.

Precision

Contents / engine sizes + md5

jv-workers/
  libNvInfer_edgellm_plugin.so   45,338,696 B   md5 17385ff85ff6e22da258eca31863867b
models/qwen3-asr-base/engines/
  llm/
    llm.engine                  550,819,820 B   md5 4f3496c3b1d2790eb5c3a4a2fe1ba465
    embedding.safetensors       311,165,016 B   md5 8db9ceda288e2470694a0bc33dbfd381
    tokenizer.json               11,429,653 B   md5 68e0da75e29b5190b2b98c2e8a49d2e2
    tokenizer_config.json, processed_chat_template.json, config.json
  audio/
    audio_encoder.engine        381,216,588 B   md5 f7a7fa8cb2b31d20e38cf664528ef03e
    config.json

Validated metrics

10-clip primed set (primed_results.json), greedy decode:

• Chinese: CER 0% (6/6 exact modulo trailing punctuation)
• English: WER ~11% (one substitution: "drop it down" -> "Drape it down")
• all clips finish_reason: end-of-sequence (no runaway / no empty output)

⚠️ rope_type must be mrope

The LLM engines/llm/config.json here carries rope_scaling.rope_type = "mrope" (mrope_section [24,20,20]). This is a load-bearing fix over the exported default ("linear"). With linear, the int4 engine emits garbage (repetition loops / empty). Do NOT regress it to linear. (config.json.bak, if present, is the pre-fix broken linear variant — delete it; do not deploy it.)

⚠️ Production decode contract (REQUIRED for correct output)

The bare engine WITHOUT priming produces garbage. You MUST drive it with:

• Sampling: temperature=0.0, top_k=1, top_p=1.0 (greedy)
• apply_chat_template=true, add_generation_prompt=false
• Assistant-turn prime — request supplies a partial assistant turn:

{
  "batch_size": 1,
  "temperature": 0.0, "top_k": 1, "top_p": 1.0,
  "max_generate_length": 256,
  "apply_chat_template": true,
  "add_generation_prompt": false,
  "requests": [{ "messages": [
    { "role": "system",    "content": "" },
    { "role": "user",      "content": [{ "type": "audio", "audio": "/path/to/mel.safetensors" }] },
    { "role": "assistant", "content": "language English<asr_text>" }
  ]}]
}

"language <Lang><asr_text>" where <Lang> ∈ {Chinese, English, ...}.

• Post-process: strip the leading "language <Lang> " from output_text (production stripLanguagePrefix).

Run command (llm_inference reference)

Production ASR runs through llm_inference / the qwen3_asr_worker. Direct reference invocation using the v0.8.0 llm_inference binary:

export EDGELLM_PLUGIN_PATH=/path/to/jv-workers/libNvInfer_edgellm_plugin.so
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

# binary lives in the build tree: edgellm-v080-build/build/examples/llm/llm_inference
./llm_inference \
  --engineDir            models/qwen3-asr-base/engines/llm \
  --multimodalEngineDir  models/qwen3-asr-base/engines/audio \
  --inputFile  input_asr.json \
  --outputFile output_asr.json

NOTE: --multimodalEngineDir points at the PARENT directory that contains the audio engine + its config.json (here engines/audio), NOT at the .engine file. Mel input is produced by tensorrt-edgellm-preprocess-audio.

Provenance

Quantization command, engine-build flags, the quantize.py:737 extra_state_dict (modelopt>=0.39) export call, and the rope fix are recorded in /home/harvest/project/qwen3-asr-ab/SOURCE_PROVENANCE.md.