qwen2audio-lora-music-descriptor

Music Descriptor

<!-- Provide a quick summary of what the model is/does. -->

Fine-tuned Model for creating musical descriptions

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

Qwen2-Audio-7B was fine-tuned using LoRA for the task of music decription

• Developed by: Kailey MacFadyen
• Language(s) (NLP): English
• Finetuned from model [optional]: Qwen2-Audio-7B

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

Enter an audio file of music and get a textual description of how it sounds.

How to Get Started with the Model

Use the code below to get started with the model.

Must load Qwen2-Audio-7B and apply this model as a PEFT adapter:

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4", # Or "fp4"
    bnb_4bit_compute_dtype="float16",
    bnb_4bit_use_double_quant=True
)
base_model = Qwen2AudioForConditionalGeneration.from_pretrained("Qwen/Qwen2-Audio-7B", quantization_config=quantization_config,
                                                        device_map="auto", trust_remote_code=True)
lora_model = PeftModel.from_pretrained(temp, "KaileyM/qwen2audio-lora-music-descriptor", device_map="auto")

processor = AutoProcessor.from_pretrained("Qwen/Qwen2-Audio-7B" ,trust_remote_code=True)

Making a description:

def describe_music(audio, model):
  prompt = "<|audio_bos|><|AUDIO|><|audio_eos|>Describe the music thoroughly:"

  inputs = processor(text=prompt, audio=audio, sampling_rate=16000, padding=False, return_tensors="pt")

  inputs = {k: v.to(model.device) for k, v in inputs.items()}

  with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=128)

  # remove prompt tokens from generation
  generated_ids = generated_ids[:, inputs["input_ids"].size(1):]

  description = processor.decode(
    generated_ids, skip_special_tokens=True)
  
  return description

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

Trained and tested on the MusicCaps dataset: https://huggingface.co/datasets/google/MusicCaps

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Training Hyperparameters

• Training regime: training_args = TrainingArguments( output_dir="./audio-qwen-lora", save_strategy="epoch", eval_strategy="epoch", logging_strategy="epoch", learning_rate = 1e-4, per_device_train_batch_size=4, per_device_eval_batch_size=4, gradient_accumulation_steps=8, num_train_epochs=3, report_to="none", load_best_model_at_end=True, fp16=True, ) <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

Evaluated using BLEU, METEOR, ROUGE, and BERT-Score to measure similarity with the original reference descriptions

Framework versions

• PEFT 0.18.1