audio-llama

Audio-LLaMA: LoRA Adapter for Audio Understanding

<img src="https://cdn-lfs-us-1.hf.co/repos/64/30/6430a20288b1b07674f3ab60cdfa7cb9483852464dfbac93f3b96b3a002959f8/948b61345b8d64ce4edff3df13d4e3a6e373e58759f20cb34ce565c2b389d27e?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27Screenshot%2525202025-03-18%252520at%2525202.26.36%2525E2%252580%2525AFPM.png%3B+filename%3D%22Screenshot%25202025-03-18%2520at%25202.26.36%25E2%2580%25AFPM.png%22%3B&response-content-type=image%2Fpng&Expires=1742329810&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc0MjMyOTgxMH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmhmLmNvL3JlcG9zLzY0LzMwLzY0MzBhMjAyODhiMWIwNzY3NGYzYWI2MGNkZmE3Y2I5NDgzODUyNDY0ZGZiYWM5M2YzYjk2YjNhMDAyOTU5ZjgvOTQ4YjYxMzQ1YjhkNjRjZTRlZGZmM2RmMTNkNGUzYTZlMzczZTU4NzU5ZjIwY2IzNGNlNTY1YzJiMzg5ZDI3ZT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSomcmVzcG9uc2UtY29udGVudC10eXBlPSoifV19&Signature=mkayensxNAMol1pHzs7mnTUzo%7E3G0gHGMULx4zPwHuMg1BD6iQOfxjtTYNFohTCAaS2XNQ%7ElWB3uxS-X4JwO2rhqVmR37jN8xtNc%7E7vHedwUd4rfTyrosOCV6wF9yVGlqSvHgQQe4aMQa9O67qb6jm2c-6I1LreuZwaWgBnAHKDW%7Em7zjH0y%7EwjFlqG%7EgzOalWJs2jwzyel7zHIhfKaozDsCxFZHuu0Y8J-4Sv%7EvOjE%7Ey8QNadYHm31-4d1KwyONWcO3dZ31ndOeLkmWjv%7EIqKhZgKDIX-L4CjZPtwHIglEhYp0dJyXbJT%7EGQYgmVuPtETT2SjwW689crMSfpj2m2A__&Key-Pair-Id=K24J24Z295AEI9" width="500">

Model Details

• Base Model: meta-llama/Llama-3.2-3B-Instruct
• Audio Model: openai/whisper-large-v3-turbo
• LoRA Rank: 32
• Task: Audio transcription from LibriSpeech dataset
• Training Framework: PEFT (Parameter-Efficient Fine-Tuning)

Usage

This is a PEFT (LoRA) adapter that needs to be combined with the base Llama model to work:

import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the LoRA configuration
config = PeftConfig.from_pretrained("cdreetz/audio-llama")

# Load the base model
model = AutoModelForCausalLM.from_pretrained(
    config.base_model_name_or_path,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)

# Load the LoRA adapter
model = PeftModel.from_pretrained(model, "cdreetz/audio-llama")

# Run inference
prompt = "Transcribe this audio:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Training

This model was fine-tuned using LoRA on audio transcription tasks. It starts with a Llama 3 base model and uses Whisper-processed audio features for audio understanding.

Limitations

This model requires special code for audio processing with Whisper before passing to the Llama model. See the Audio-LLaMA repository for full usage instructions.