Qwen2-VL-7B-Speech-LoRA

Qwen2-VL-7B-Speech-LoRA

This repository contains LoRA adapters only (~700 MB), NOT the full model.

You must load the base model DanJZY/Qwen2-VL-7B-Speech first, then apply these adapters on top.

What's in this repo

• LoRA adapters for the LLM decoder layers (28 layers × 7 projections = 196 modules, rank 64)
• Updated audio projector weights (via PEFT modules_to_save)
• Total adapter size: ~700 MB

Usage

import torch
from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info

# Step 1: Load the base model (required)
base_model = Qwen2VLForConditionalGeneration.from_pretrained(
    "DanJZY/Qwen2-VL-7B-Speech",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)

# Step 2: Apply LoRA adapters from this repo
model = PeftModel.from_pretrained(base_model, "DanJZY/Qwen2-VL-7B-Speech-LoRA")
model.eval()

processor = Qwen2VLProcessor.from_pretrained("DanJZY/Qwen2-VL-7B-Speech")

# Step 3: Transcribe audio
messages = [
    {"role": "user", "content": [
        {"type": "audio", "audio": "path/to/audio.wav"},
        {"type": "text", "text": "Transcribe this audio."},
    ]},
]

image_inputs, video_inputs, audio_inputs = process_vision_info(messages)
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
batch = processor(text=[text], audios=audio_inputs, return_tensors="pt", padding=True)
batch = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}

with torch.inference_mode():
    output_ids = model.generate(
        **batch, max_new_tokens=256, num_beams=2,
        do_sample=False, repetition_penalty=1.1,
    )
prompt_len = batch["input_ids"].shape[1]
print(processor.batch_decode(output_ids[:, prompt_len:], skip_special_tokens=True)[0])

Results

Evaluated on speechbrain/LargeScaleASR test set (8,087 samples):

Training details

• Method: LoRA (rank 64, alpha 128, dropout 0.05) on all LLM decoder attention + MLP projections
• Data: speechbrain/LargeScaleASR small split (~107K training samples, 3 epochs)
• Hardware: 6× NVIDIA RTX 6000 Ada (49 GB each), ~5.5 hours
• Learning rate: 2e-5, cosine schedule, 3% warmup
• Effective batch size: 96 (2/gpu × 8 gradient accumulation × 6 GPUs)

Dependencies

Requires our forked transformers with audio support. See the training repository for setup instructions.