Downloads · 30 days
6
10% of all-time downloads
Amirhossein75/qwen2-vl-2b-mmhs150k-lora
qwen2-vl-2b-mmhs150k-lora is a text generation model from Amirhossein75. Use it when you need the model to write or continue text. It is set up for peft.
This is a LoRA adapter and cannot be loaded directly with AutoModel. Load it as follows:
Downloads · 30 days
6
10% of all-time downloads
All-time downloads
63
Public
Repo size
18.5 MB
Likes
0
Public
Click a slice to open those files.
.safetensors18.5 MB · 100%
From the Hugging Face model README
This is a LoRA adapter and cannot be loaded directly with AutoModel. Load it as follows:
from transformers import Qwen2VLForConditionalGeneration
from peft import PeftModel
# Load base model
base_model = Qwen2VLForConditionalGeneration.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")
# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "Amirhossein75/qwen2-vl-2b-mmhs150k-lora")
This repository contains a LoRA adapter fine-tuned on MMHS150K (Multi-Modal Hate Speech) for multi-label hateful content detection from paired text + image inputs.
The approach follows the project at https://github.com/amirhossein-yousefi/text_image_multi_modal_vlm: instead of training a classification head, the model is prompted to generate a strict JSON array of labels, which is then parsed and scored as multi-label predictions.
racist, sexist, homophobe, religion, otherhateimport torch
from PIL import Image
from peft import PeftModel
from transformers import AutoProcessor, Qwen2VLForConditionalGeneration
base_id = "Qwen/Qwen2-VL-2B-Instruct"
adapter_id = "Amirhossein75/qwen2-vl-2b-mmhs150k-lora" # this repo
processor = AutoProcessor.from_pretrained(base_id)
model = Qwen2VLForConditionalGeneration.from_pretrained(
base_id,
torch_dtype=torch.float16,
device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)
image = Image.open("path/to/image.jpg").convert("RGB")
text = "Some text to analyze"
labels = ["racist", "sexist", "homophobe", "religion", "otherhate"]
system = "Return JSON only."
user = (
"Given the image and text, return a JSON array containing zero or more of these labels: "
+ ", ".join([f"\"{l}\"" for l in labels])
)
messages = [
{"role": "system", "content": system},
{"role": "user", "content": [
{"type": "text", "text": user + "\n\nText: " + text},
{"type": "image", "image": image},
]},
]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[image], return_tensors="pt", padding=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=64)
print(processor.decode(out[0], skip_special_tokens=True))
text, image_path, labels and an images/ directory.r=4, lora_alpha=32, lora_dropout=0.05, target modules q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_projAs reported in the associated training/evaluation repository (see link above), the Qwen2-VL + LoRA/QLoRA runs were trained on:
nvidia-smi)Metrics follow the associated code repo: multi-label scores computed from generated JSON labels.
0.6172, macro F1 0.5077, subset accuracy 0.4366, hamming loss 0.142760.6110, macro F1 0.4992