Downloads · 30 days
5
13% of all-time downloads
ahczhg/qwen3-vl-2b-pagoda-archaeology-lora
qwen3-vl-2b-pagoda-archaeology-lora is a image-text-to-text model from ahczhg. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
<div align="center" <img src="https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct/resolve/main/qwen2vllogo.png" width="400"/ </div
Downloads · 30 days
5
13% of all-time downloads
All-time downloads
40
Public
Repo size
53.7 MB
Likes
1
Public
Click a slice to open those files.
.safetensors42.2 MB · 73%
From the Hugging Face model README
This model is a fine-tuned version of Qwen/Qwen2-VL-2B-Instruct using LoRA (Low-Rank Adaptation) on the Pagoda text-and-image dataset.
This is a multimodal vision-language model that has been fine-tuned to understand and describe images with a focus on pagoda-related content. The model uses LoRA adapters applied to both the vision encoder and language decoder, enabling efficient fine-tuning while maintaining high performance.
qkv (attention projections in vision transformer)q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_projpip install transformers accelerate peft pillow torch
# For 4-bit quantization (optional but recommended):
pip install bitsandbytes
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from PIL import Image
import torch
# Load base model with 4-bit quantization
model = Qwen2VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2-VL-2B-Instruct",
device_map="auto",
trust_remote_code=True,
torch_dtype=torch.bfloat16
)
# Load LoRA adapter
model = PeftModel.from_pretrained(model, "ahczhg/qwen3-vl-2b-pagoda-lora")
model.eval()
# Load processor
processor = AutoProcessor.from_pretrained(
"ahczhg/qwen3-vl-2b-pagoda-lora",
trust_remote_code=True
)
# Load and process image
image = Image.open("your_image.jpg")
conversation = [
{
"role": "user",
"content": [
{"type": "image", "image": image},
{"type": "text", "text": "Describe this image in detail."}
]
}
]
# Generate response
text = processor.apply_chat_template(
conversation,
tokenize=False,
add_generation_prompt=True
)
inputs = processor(
text=[text],
images=[[image]],
return_tensors="pt"
).to(model.device)
with torch.no_grad():
output_ids = model.generate(
**inputs,
max_new_tokens=256,
do_sample=True,
temperature=0.7,
top_p=0.9
)
response = processor.batch_decode(
output_ids,
skip_special_tokens=True,
clean_up_tokenization_spaces=False
)[0]
print(response)
For faster inference, you can merge the LoRA weights into the base model:
from peft import PeftModel
# Load and merge
model = PeftModel.from_pretrained(base_model, "ahczhg/qwen3-vl-2b-pagoda-lora")
merged_model = model.merge_and_unload()
# Save merged model
merged_model.save_pretrained("./merged_model")
processor.save_pretrained("./merged_model")
Input: [Image of a pagoda]
Prompt: "Describe this image in detail."
Output: [Model-generated description]
This model inherits the license from the base model:
@misc{qwen2vl-pagoda-lora,
author = {Your Name},
title = {Qwen2-VL-2B Fine-tuned on Pagoda Dataset},
year = {2025},
publisher = {HuggingFace},
howpublished = {\url{https://huggingface.co/ahczhg/qwen3-vl-2b-pagoda-lora}}
}