qwen3.5-2b-vlm-weld-explainability-lora

Model Card for Model ID

A LoRA adapter fine-tuned on top of unsloth/Qwen3.5-2B to provide explainable weld defect analysis. Given a weld image and accompanying sensor telemetry (weld current, voltage, pressure, CO2 flow, feed rate, wire consumed), the model produces a structured report: weld quality classification (1 of 12 classes), visual observation, sensor analysis, model confidence, defect probability, severity, root cause, and corrective actions.

Model Details

Model Description

This is a PEFT/LoRA adapter fine-tuned from the unsloth/Qwen3.5-2B vision-language base model, trained on the IntelLabs/Intel_Robotic_Welding_Multimodal_Dataset (gated, publicly accessible dataset). Given a weld image paired with a natural-language prompt containing sensor telemetry, the model returns a structured, human-readable explainability report covering:

• Weld quality/defect classification (1 of 12 classes)
• Visual observation summary
• Sensor analysis output
• Model confidence
• Defect probability
• Severity assessment
• Root cause analysis
• Corrective action recommendations

The adapter is intended to be served alongside the base model (e.g., via vLLM with --enable-lora) rather than used standalone.

• Fine tuned by: Intel (internal team)
• Model type: LoRA adapter for a vision-language model (image + text → text), supervised fine-tuned (SFT) for weld defect explainability
• Language(s) (NLP): English
• License: Apache 2.0
• Finetuned from model [optional]: unsloth/Qwen3.5-2B

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

For direct use of how this fine tuned LoRA adapter is being used, one can refer to link

Downstream Use

Load the base model unsloth/Qwen3.5-2B together with this LoRA adapter and query it with a weld image plus a text prompt that includes sensor telemetry (e.g., primary weld current, secondary weld voltage, pressure, CO2 weld flow, feed rate, wire consumed). The model returns a structured weld quality report for use by weld inspectors, quality engineers, and robotic welding operators as a decision-support / explainability aid — it is not intended to autonomously accept/reject welds without human review.

Out-of-Scope Use

• Not intended for weld processes, materials, or defect types outside the 12 classes covered by the training dataset.
• Not intended as the sole/automated safety-critical decision-maker for accepting or rejecting welds without qualified human review.
• Not intended for general-purpose visual question answering or use cases unrelated to robotic MIG/MAG/TIG weld inspection.
• Not validated on sensor telemetry ranges/units outside those present in the training data.

Bias, Risks, and Limitations

• Trained on a specific robotic welding dataset (IntelLabs Intel Robotic Welding Multimodal Dataset); performance may degrade on welds captured with different cameras, lighting, materials, or welding equipment/sensors than those represented in training.
• As with any LLM/VLM, the model can hallucinate root-cause explanations or corrective actions that sound plausible but are not grounded in the actual sensor data or image.
• Class imbalance in the underlying dataset (if present) may bias predictions toward more frequently represented defect classes.
• Evaluation results are not yet available (see Evaluation section), so quantitative accuracy/robustness claims cannot currently be made.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model. The example below serves the base model with the LoRA adapter via vLLM and queries it through an OpenAI-compatible client. Replace the image path and sensor telemetry with your own data.

Serve with vLLM Docker (base model + LoRA adapter):

$> docker run -t -d \
  --shm-size 4g \
  --net=host \
  --ipc=host \
  --privileged \
  -v /dev/dri/by-path:/dev/dri/by-path \
  -v /path/.cache/huggingface:/root/.cache/huggingface \
  -v /fine-tuning:/fine-tuning \
  --name=vllm-server \
  --device /dev/dri:/dev/dri \
  --entrypoint=/bin/bash \
  intel/vllm:0.21.0-ubuntu24.04-20260625

$> docker exec -it vllm-server bash

$> VLLM_WORKER_MULTIPROC_METHOD=spawn vllm serve unsloth/Qwen3.5-2B   \
  --dtype=float16 \
  --enforce-eager \
  --port 8005 \
  --block-size 32 \
  --gpu-memory-utilization 0.182 \
  --no-enable-prefix-caching \
  --trust-remote-code \
  --max-num-batched-tokens=8192 \
  --max-model-len 8192 \
  --served-model-name unsloth/Qwen3.5-2B \
  --enable-lora \
  --lora-modules qwen3.5-2b-adapter=/fine-tuning/qwen_3.5_2b_adapter/checkpoint/ \
  -tp=1 \
  --quantization fp8 \
  --attention-backend TRITON_ATTN

Query with an OpenAI-compatible client:

import base64
from openai import OpenAI

client = OpenAI(
    api_key="sk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    base_url="http://<HOST>:8005/v1"
)

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

b64_image = encode_image("path/to/weld_image.jpg")

messages = [
    {
        "role": "system",
        "content": [{
            "type": "text",
            "text": (
                "You are an expert weld quality inspector and metallurgical engineer "
                "with deep knowledge of MIG/MAG/TIG arc welding processes and industrial "
                "weld defect analysis per AWS D1.1 and ISO 5817 standards. When shown a "
                "weld image alongside time-series sensor readings, classify the weld "
                "quality, identify any defect type, explain the root cause using sensor "
                "evidence, assess severity, and recommend corrective actions."
            ),
        }],
    },
    {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64_image}"}},
            {
                "type": "text",
                "text": (
                    "Given this weld image and the sensor telemetry, produce a structured "
                    "weld quality report covering defect classification, root cause, and "
                    "remediation steps.\nSensor Data:\n"
                    "  • Primary Weld Current: <value> A\n"
                    "  • Secondary Weld Voltage: <value> V\n"
                    "  • Pressure: <value> bar\n"
                    "  • CO2 Weld Flow: <value> L/min\n"
                    "  • Feed: <value> mm/min\n"
                    "  • Wire Consumed: <value> mm"
                ),
            },
        ],
    },
]

response = client.chat.completions.create(
    model="qwen3.5-2b-adapter", # alias set in vLLM docker serve.
    messages=messages,
    max_tokens=2048,
    temperature=1.5,
    extra_body={"min_p": 0.1},
)
print(response)

Example (illustrative placeholder) output format:

Weld Classification: <one of 12 defect classes>
Visual Observation: <description of what is visually apparent in the image>
Sensor Analysis: <interpretation of the provided telemetry values>
Model Confidence: <confidence score>
Defect Probability: <probability score>
Severity: <low / medium / high>
Root Cause: <inferred cause based on visual + sensor evidence>
Corrective Actions: <recommended remediation steps>

(A real sample image, prompt, and generated output will be added here once available.)

Training Details

Training Data

IntelLabs/Intel_Robotic_Welding_Multimodal_Dataset — a gated (but publicly accessible upon request) multimodal dataset pairing robotic weld images with time-series sensor telemetry (weld current, voltage, pressure, CO2 flow, feed rate, wire consumed) across 12 weld quality/defect classes, along with conversational (system/user/assistant) annotations used for supervised fine-tuning.

Training Procedure

Fine-tuned using Unsloth's FastVisionModel with 4-bit quantized loading and gradient checkpointing, wrapped with a trl.SFTTrainer / SFTConfig supervised fine-tuning loop. LoRA was applied to vision layers, language layers, attention modules, and MLP modules.

Training Hyperparameters

• Training regime: 4-bit quantized base model (load_in_4bit=True) with LoRA adapter trained in default (non-mixed) precision via Unsloth/TRL; inference server uses float16. <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
• LoRA configuration: r=16, lora_alpha=16, lora_dropout=0, bias="none", target modules = vision layers + language layers + attention modules + MLP modules, use_rslora=False, random_state/seed=3407
• Optimizer: AdamW (adamw_torch on CPU/XPU; adamw_8bit on CUDA), weight_decay=0.001
• Learning rate: 2e-4, linear LR scheduler, 5 warmup steps
• Batch size: per-device train/eval batch size = 4, gradient accumulation steps = 4 (effective batch size = 16)
• Sequence length: max_length = 2048
• Steps: trained to checkpoint-432 (final adapter used for serving)
• Eval/save strategy: evaluation and checkpoint saving every 20 steps

Speeds, Sizes, Times [optional]

Check if this number needs to be published

• Inference hardware: Intel Panther Lake (PTL) 365H
• Inference throughput: ~27 tokens/sec generation via vLLM (LoRA adapter served alongside base model, --enable-lora)

Evaluation

Formal evaluation results (accuracy/F1 on the 12 weld defect classes, qualitative review of generated explanations, etc.) are to be added.

Technical Specifications [optional]

Model Architecture and Objective

LoRA adapter (r=16, alpha=16) applied to the vision, language, attention, and MLP modules of the unsloth/Qwen3.5-2B vision-language model, trained with a supervised fine-tuning (next-token prediction on assistant responses) objective to produce structured weld-defect explainability reports from image + sensor-telemetry-augmented text prompts.

Compute Infrastructure

Hardware

• Fine-tuning: Intel Core Ultra 7 255H (ARL), Intel Xeon + Intel Arc B580
• Inference: Intel Panther Lake (PTL), served via vLLM with Intel XPU support

Software

• Unsloth (FastVisionModel, UnslothVisionDataCollator)
• TRL (SFTTrainer, SFTConfig)
• Hugging Face transformers
• PEFT 0.19.1
• vLLM (XPU build) for serving with --enable-lora
• PyTorch (with Intel XPU support)

Glossary [optional]

• LoRA: Low-Rank Adaptation, a parameter-efficient fine-tuning technique.
• PEFT: Parameter-Efficient Fine-Tuning.
• SFT: Supervised Fine-Tuning.
• ARL: Arrow Lake (Intel Core Ultra Series 2 processor family).
• PTL: Panther Lake (Intel Core Ultra Series 3 processor family).

Framework versions

• PEFT 0.19.1