qwen-color-pattern-lora

qwen-color-pattern-lora

A LoRA adapter for Qwen2-VL-2B-Instruct that predicts primary, secondary, and tertiary colors plus surface pattern from a garment/fabric image.

Designed to be used alongside PrathamMMallya99/textilenet-qwen-lora — both adapters load on the same base model, each handling its own task.

---

Output

{
  "fibre_type":      "cotton",
  "primary_color":   "cobalt blue",
  "secondary_color": "white",
  "tertiary_color":  null,
  "pattern":         "stripe"
}

---

Install

pip install "transformers>=4.47.0" "peft>=0.13.0" "accelerate>=0.34.0" \
            "qwen-vl-utils" "pillow" torch

---

Quick start — dual adapter inference

import json
import torch
from PIL import Image
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info   # required for Qwen2-VL image encoding

IMAGE_PATH = "/path/to/your/fabric_image.jpg"   

MODEL_ID   = "Qwen/Qwen2-VL-2B-Instruct"
IMG_PX     = 224 * 224
DEVICE     = "cuda" if torch.cuda.is_available() else "cpu"

print("Loading base model (~4GB, downloads once)...")
base = Qwen2VLForConditionalGeneration.from_pretrained(
    MODEL_ID,
    torch_dtype = torch.float16 if DEVICE == "cuda" else torch.float32,
    device_map  = "auto",
)

model = PeftModel.from_pretrained(
    base, "PrathamMMallya99/textilenet-qwen-lora", adapter_name="fibre"
)
model.load_adapter("PrathamMMallya99/qwen-color-pattern-lora", adapter_name="color")
model.eval()

processor = AutoProcessor.from_pretrained(MODEL_ID, min_pixels=IMG_PX, max_pixels=IMG_PX)
print("Both adapters ready.")

def _run(adapter_name: str, image: Image.Image, prompt: str, max_new_tokens: int) -> str:
    model.set_adapter(adapter_name)
    messages = [{"role": "user", "content": [
        {"type": "image", "image": image},
        {"type": "text",  "text": prompt},
    ]}]
    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    image_inputs, video_inputs = process_vision_info(messages)
    inputs = processor(
        text=[text], images=image_inputs, videos=video_inputs,
        return_tensors="pt", padding=True,
    )
    inputs = {k: v.to(DEVICE) for k, v in inputs.items()}
    with torch.no_grad():
        out = model.generate(
            **inputs, max_new_tokens=max_new_tokens, do_sample=False,
            pad_token_id=processor.tokenizer.eos_token_id,
        )
    return processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()


image = Image.open(IMAGE_PATH).convert("RGB").resize((448, 448), Image.LANCZOS)

# Pass 1 — fibre type (textilenet LoRA)
fibre_raw = _run(
    "fibre", image,
    "Identify the textile fibre type of the fabric in this image.\n"
    "Reply with only the fibre name — no explanation, no punctuation.",
    max_new_tokens=20,
)

# Pass 2 — color + pattern (color LoRA)
color_raw = _run(
    "color", image,
    "You are a fashion color and textile expert.\n"
    "Identify the dominant colors and surface pattern of the fabric.\n"
    "Use precise fashion terms: prefer 'cobalt blue' over 'blue', 'dusty rose' over 'pink'.\n"
    "Reply with ONLY a JSON object — no markdown, no extra text:\n"
    '{"primary_color":"<color>","secondary_color":"<color or null>",'
    '"tertiary_color":"<color or null>","pattern":"<pattern>"}',
    max_new_tokens=100,
)

try:
    color_data = json.loads(color_raw)
except json.JSONDecodeError:
    s, e = color_raw.find("{"), color_raw.rfind("}")
    color_data = json.loads(color_raw[s:e+1]) if s != -1 else {}

result = {
    "fibre_type":      fibre_raw.lower().rstrip("."),
    "primary_color":   color_data.get("primary_color"),
    "secondary_color": color_data.get("secondary_color"),
    "tertiary_color":  color_data.get("tertiary_color"),
    "pattern":         color_data.get("pattern"),
}

print(result)
# {'fibre_type': 'cotton', 'primary_color': 'cobalt blue',
#  'secondary_color': 'white', 'tertiary_color': None, 'pattern': 'stripe'}

---

Color vocabulary (100+ fashion colors)

black · off black · charcoal · dark grey · grey · light grey · silver · off white · white · ivory · cream · grey melange · brown · dark brown · coffee brown · chocolate · tan · camel · sand · taupe · mushroom brown · khaki · nude · skin · beige · red · bright red · dark red · maroon · burgundy · wine · rust · brick · coral · tomato · blush · rose · dusty rose · mauve · pink · hot pink · magenta · fuchsia · orange · burnt orange · peach · apricot · yellow · mustard · gold · copper · bronze · green · dark green · forest green · bottle green · sea green · mint · sage · olive · army green · lime green · teal · turquoise · jade · light blue · sky blue · cornflower · blue · cobalt blue · royal blue · navy blue · midnight blue · denim · powder blue · lavender · lilac · violet · purple · dark purple · plum · indigo · metallic

Pattern vocabulary

solid · stripe · check · plaid · floral · printed · geometric · abstract · animal print · camouflage · polka dot · houndstooth · paisley · tie-dye · ombre · embroidered · textured · knit · lace · jacquard · graphic · patchwork

---

Architecture

Qwen2-VL-2B-Instruct (base, frozen)
  ├── adapter "fibre"  — textilenet-qwen-lora  (r=64, trained on 33-class fibre dataset)
  └── adapter "color"  — qwen-color-pattern-lora  (r=16, trained on DeepFashion + fashion-product-images-small)

• Color matching uses CIELAB perceptual distance (not RGB) — so navy blue and cobalt blue are correctly separated
• Pattern detection uses FFT periodicity on pixel gradients — detects stripes/checks visually rather than from text labels
• Both adapters live in GPU memory simultaneously switching between them is zero-cost

---

Related

• Fibre adapter: PrathamMMallya99/textilenet-qwen-lora
• Base model: Qwen/Qwen2-VL-2B-Instruct