Downloads · 30 days
0
ahczhg/qwen2-vl-2b-pagoda-lora
qwen2-vl-2b-pagoda-lora is a machine learning model from ahczhg. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Fine-tuning the Qwen2-VL-2B-Instruct multimodal vision-language model on the Pagoda dataset using LoRA (Low-Rank Adaptation).
Downloads · 30 days
0
Access
Public
Updated Nov 16, 2025
Repo size
—
Likes
1
Public
Click a slice to open those files.
.ipynb40.8 KB · 59%
From the Hugging Face model README
Fine-tuning the Qwen2-VL-2B-Instruct multimodal vision-language model on the Pagoda dataset using LoRA (Low-Rank Adaptation).
This project demonstrates efficient fine-tuning of a state-of-the-art multimodal model using:
qwen2-vl-pagoda-lora/
├── qwen3_vl_lora_finetuning.ipynb # Main training notebook
├── README.md # This file
├── LICENSE # Apache 2.0 License
├── .gitignore # Git ignore file
└── qwen3-vl-2b-pagoda-lora/ # Output directory (after training)
├── adapter_config.json # LoRA configuration
├── adapter_model.safetensors # LoRA weights
├── README.md # Model card
└── ... # Other model files
| Parameter | Value |
|---|---|
| Base Model | Qwen2-VL-2B-Instruct |
| Dataset | 1000 samples from Pagoda dataset |
| Train/Val Split | 900 / 100 (90% / 10%) |
| LoRA Rank | 8 |
| LoRA Alpha | 16 |
| Batch Size | 1 (effective: 8 with gradient accumulation) |
| Learning Rate | 2e-4 |
| Epochs | 1 |
| Optimizer | PagedAdamW 8-bit |
| Precision | bfloat16 |
| Training Time | ~15-20 minutes |
git clone https://github.com/YOUR_USERNAME/qwen2-vl-pagoda-lora.git
cd qwen2-vl-pagoda-lora
pip install -r requirements.txt
Or install manually:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install transformers accelerate datasets peft bitsandbytes pillow qwen-vl-utils
huggingface-cli login
jupyter notebook qwen3_vl_lora_finetuning.ipynb
After training, use the fine-tuned model:
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from PIL import Image
# Load model
base_model = Qwen2VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2-VL-2B-Instruct",
device_map="auto",
trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, "./qwen3-vl-2b-pagoda-lora")
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct", trust_remote_code=True)
# Process image
image = Image.open("your_image.jpg")
conversation = [{
"role": "user",
"content": [
{"type": "image", "image": image},
{"type": "text", "text": "Describe this image."}
]
}]
text = processor.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[[image]], return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256)
print(processor.batch_decode(output, skip_special_tokens=True)[0])
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
The base model (Qwen2-VL-2B-Instruct) is also licensed under Apache 2.0.
If you use this project in your research, please cite:
@misc{qwen2vl-pagoda-lora,
author = {Your Name},
title = {Fine-tuning Qwen2-VL-2B on Pagoda Dataset with LoRA},
year = {2025},
publisher = {GitHub},
howpublished = {\url{https://github.com/YOUR_USERNAME/qwen2-vl-pagoda-lora}}
}
Contributions are welcome! Please feel free to submit a Pull Request.
For questions or issues, please open an issue on GitHub or contact [your email].
Note: This is a demonstration project for educational purposes. The model was trained on a limited dataset (1000 samples) for 1 epoch. For production use, consider training on more data for multiple epochs.