listen-on-demand-adapters

Listen-on-Demand LoRA adapters for Qwen3-Omni-30B-A3B-Instruct

Three LoRA adapters that teach Qwen3-Omni-30B-A3B-Instruct when to inspect the audio of a video through a text-protocol tool and what to do with the tool's output. They come from the Listen-on-Demand project (sequel to When Vision Speaks for Sound, arXiv:2605.16403); the full project — data, code, evaluation rows, documents — is in Rakancorle1/listen-on-demand. Project status: paused 2026-09-18.

Each folder holds final/ (the adapter used in every reported number), the two epoch checkpoints (checkpoint-*/, with optimizer/scheduler/RNG state for resuming), log_history.json and train_manifest.json (exact row composition, args, world size). v1_1/final is the reference adapter by the PI's ruling of 2026-09-08; v1_2/final is better on 8 of the 12 headline cells and whether it replaces v1.1 is an open decision (archive docs/STATUS.md §5).

Training configuration (all three)

LoRA r = 16, α = 32, dropout 0.05 on the thinker text decoder's attention projections only — ^model\.layers\.\d+\.self_attn\.(q|k|v|o)_proj$, 192 modules, 13.4 M trainable parameters; audio tower, vision encoder and the MoE experts untouched. Loss on assistant tokens through <|im_end|>. lr 1e-4 cosine, 2 epochs, per-device batch 1 (variable-shape multimodal inputs) × grad-accum 4 × 8 H200 (one full model copy per GPU, data-parallel). PEFT 0.18.1, trainer sft/train_lora.py in the archive. Prompts are byte-identical to the evaluation runner's.

adapter_config.json has an empty base_model_name_or_path because training loaded the base model from a local path — pass the base model explicitly when loading.

Loading

from transformers import Qwen3OmniMoeForConditionalGeneration, Qwen3OmniMoeProcessor
from peft import PeftModel

base = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
model = Qwen3OmniMoeForConditionalGeneration.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
processor = Qwen3OmniMoeProcessor.from_pretrained(base)
# the adapter targets the thinker sub-module
model.thinker = PeftModel.from_pretrained(model.thinker, "Rakancorle1/listen-on-demand-adapters", subfolder="v1_2/final")
model.thinker = model.thinker.merge_and_unload()

The project's own runner does exactly this and also implements the protocol: python sft/run_agentic_baseline.py --adapter <path to v1_2/final> ... (archive repo). The protocol is a text protocol, not API function calling: round 1 = video + audio + TOOL_INSTRUCTION + question → the model either answers or replies with the single line CALL_AUDIO_TOOL; round 2 = the tool's evidence string (RMS silence gate → silero-VAD → whisper-large-v3 transcript → AST top-5 tags; "No speech transcribed." when the gated transcript is empty) + "Now answer the question". Train and eval must use the same evidence wording (--legacy-evidence reproduces the v1-era string).

Results (identical runner, Claude Opus 5 grader; accuracy %, baseline → v1.1 → v1.2)

original = no false alarm on untouched audio (hedge_ok criterion = the training filter's question), mute = says silent, swap = says mismatch. Oops/FailArmy = THUD-Eval, a different source from the training clips; n = 142 per cell.

• Tool-call rate on questions that do not mention sound (implicit): 14–38 % at baseline → 100 % for every adapter, while visual-only questions stay at 0–3 % — the decision became a question-type decision, not "always call".
• Guard cell (vision_dominant answers must not raise an audio conflict): 99.8–100 % for every adapter.
• Cost: false alarms on untouched Oops audio 1.2 % → 11.8 / 11.7 / 12.0 % (v1 / v1.1 / v1.2); ~78 % are faithful reports of tool output that lacked the expected sound, the rest genuine over-suspicion. This column did not move in three rounds.
• Oops swap is a corpus ceiling (within-compilation swaps; every adapter 27–34 %); class-disjoint same-source swaps reach 80 %.

Licence

Adapter weights: Apache 2.0 (same as the base model). The training text was written by Claude Opus 5 / Sonnet 5 and by the model itself over VGG-SS clips; see the archive repo for provenance.