audsemthinker-qa-grpo

AudSemThinker-QA-GRPO

Corresponding paper: https://arxiv.org/abs/2505.14142

Model Description

AudSemThinker-QA-GRPO is an advanced variant of AudSemThinker, fine-tuned using Group Relative Policy Optimization (GRPO) with Verifiable Rewards (RLVR). This approach enhances reasoning capabilities and allows for controlled thinking budget during generation. It leverages the structured reasoning framework of AudSemThinker (thinking, semantic elements, answer phases) but is specifically optimized for multiple-choice audio question answering. This model is designed to produce accurate answers while maintaining a controlled reasoning length in its <think> section.

How to Use

To use AudSemThinker-QA-GRPO for audio question answering, you can load it using the transformers library. Ensure you have torch, torchaudio, and soundfile installed.

import soundfile as sf
from transformers import Qwen2_5OmniThinkerForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info
import torchaudio

# default: Load the model on the available device(s)
model = Qwen2_5OmniThinkerForConditionalGeneration.from_pretrained(
    "gijs/audsemthinker", 
    torch_dtype="auto", 
    device_map="auto",
    trust_remote_code=True
)

# We recommend enabling flash_attention_2 for better acceleration and memory saving.
# model = Qwen2_5OmniThinkerForConditionalGeneration.from_pretrained(
#     "gijs/audsemthinker",
#     torch_dtype="auto",
#     device_map="auto",
#     attn_implementation="flash_attention_2",
#     trust_remote_code=True
# )

processor = Qwen2_5OmniProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B", trust_remote_code=True)

# Load and preprocess audio
audio_file = "path/to/your/audio.wav"
audio_input, sampling_rate = torchaudio.load(audio_file)
if sampling_rate != processor.feature_extractor.sampling_rate:
    audio_input = torchaudio.transforms.Resample(
        orig_freq=sampling_rate, 
        new_freq=processor.feature_extractor.sampling_rate
    )(audio_input)
audio_input = audio_input.squeeze().numpy()

# Example multiple-choice question
question = "What type of sound is present in the audio? Options: (A) Speech (B) Music (C) Environmental Sound (D) Silence"
user_prompt_text = f"You are given a question and an audio clip. Your task is to answer the question based on the audio clip. First, think about the question and the audio clip and put your thoughts in <think> and </think> tags. Then reason about the semantic elements involved in the audio clip and put your reasoning in <semantic_elements> and </semantic_elements> tags. Then answer the question based on the audio clip, put your answer in <answer> and </answer> tags.\nQuestion: {question}"

# Conversation format
conversation = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."}
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": audio_input},
            {"type": "text", "text": user_prompt_text}
        ],
    },
]

# Preparation for inference
text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
audios, images, videos = process_mm_info(conversation, use_audio_in_video=False)
inputs = processor(
    text=text, 
    audio=audios, 
    images=images, 
    videos=videos, 
    return_tensors="pt", 
    padding=True
)
inputs = inputs.to(model.device).to(model.dtype)

# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=512)
response = processor.batch_decode(output_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
print(response[0])

# Expected output format for QA:
# <think>...detailed reasoning about the audio scene and question...</think>
# <semantic_elements>...list of identified semantic descriptors...</semantic_elements>
# <answer>...selected option (e.g., (B) Music)...</answer>

Training Data

AudSemThinker-QA-GRPO is fine-tuned on the multiple-choice Question Answering (QA) subset of the AudSem dataset (approximately 140k examples). This subset provides easily verifiable correct answers, making it suitable for Reinforcement Learning with Verifiable Rewards (RLVR).

Training Procedure

• Base Model: Qwen2.5-Omni-7B.
• Fine-tuning Paradigm: Reinforcement Learning with Group Relative Policy Optimization (GRPO).
• Reward Functions:
  • Accuracy Reward: Evaluates the correctness of the content within the <answer> tags using string matching for multiple-choice questions.
  • Format Adherence Reward: Encourages strict adherence to the prescribed XML-tag structure (<think>, <semantic_elements>, <answer>), checking for presence, correct order, and proper encapsulation.
  • Length Constraint Reward: Specifically targets the <think> phase, penalizing deviations from a target thinking length (25 words for this model) to promote controlled reasoning budget.
• Parameter-Efficient Fine-tuning: LoRA (Low-Rank Adaptation).
• GRPO Loss Type: Default with beta = 0.01.
• Generations per prompt (k): 6.
• Precision: bf16.
• Batch Size: 2 per device.
• Hardware: Trained on four H100 GPUs, utilizing DeepSpeed ZeRO-3 and vLLM for efficient training and inference.
• Training Time: Approximately 10 hours.

Evaluation Results

AudSemThinker-QA-GRPO demonstrates strong performance on multiple-choice QA tasks, showcasing the effectiveness of GRPO in guiding the model towards desired reasoning patterns and controlled thinking lengths.

Limitations and Bias

• Generalization to Open-Ended Tasks: While GRPO is effective for multiple-choice QA, its performance on open-ended tasks like general audio captioning or free-form QA may not always surpass SFT, as verifying the quality of longer, more subjective generated text is more challenging for automated reward models.
• Thinking Budget Sensitivity: The effectiveness of the length constraint reward can depend on parameters (alpha, delta) and the initial average output length of the model. Excessively long target reasoning phases, if they fall outside the effective reward range, may not translate to better performance under the current setup.
• Data Contamination: While AudSem is designed to minimize overlap, the underlying Qwen2.5-Omni pretrained model might have encountered data present in test sets during its initial pretraining.

Ethical Considerations

• Data Sourcing: The AudSem dataset is primarily sourced from YouTube closed captions. While systematic checks for harmful content (e.g., child abuse, hate speech, sexual content, harassment) were performed and YouTube's community guidelines provide a safeguard, inherent biases or problematic content from the original video sources could potentially be present.
• Societal Impact: AudSemThinker-QA-GRPO can contribute to positive societal impacts by enhancing audio-language understanding, particularly in scenarios requiring precise and controlled question answering from audio, potentially leading to more reliable automated systems.

Citation

@misc{wijngaard2025audsemthinkerenhancingaudiolanguagemodels,
  title={AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound}, 
  author={Gijs Wijngaard and Elia Formisano and Michele Esposito and Michel Dumontier},
  year={2025},
  eprint={2505.14142},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2505.14142}, 
}