light-vlm-0.6b

Light VLM 0.6B

Light VLM is a lightweight multimodal vision-language model built upon Qwen2.5-0.5B-Instruct and CLIP ViT-base-patch16.

Demonstration

Below are some examples demonstrating the model's capabilities on various vision-language tasks.

Image Captioning

Model Architecture

• Language Model: Qwen2.5-0.5B-Instruct
• Vision Encoder: openai/clip-vit-base-patch16
• Projector: MLP (768 → 2048 → 896)
• Fine-tuning: LoRA (rank=64, alpha=128)

Total Parameters

~590M (0.6B)

Features

• Image captioning
• Visual question answering
• Multi-device support (CUDA/MPS/CPU)
• OpenAI-compatible API

Quick Start

from PIL import Image
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import CLIPVisionModel, CLIPImageProcessor

# Load model components
model_name = "brightxiaolei/light-vlm-0.6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
vision_encoder = CLIPVisionModel.from_pretrained(f"{model_name}/vision_model")
image_processor = CLIPImageProcessor.from_pretrained(f"{model_name}/vision_model")

# Example: Image captioning
image = Image.open("example.jpg")
inputs = image_processor(images=image, return_tensors="pt")
pixel_values = inputs['pixel_values']

API Usage

curl -X POST "http://localhost:8000/v1/chat/completions" \\
  -H "Content-Type: application/json" \\
  -d '{
    "model": "light-vlm-0.6b",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "描述这张图片"},
          {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
        ]
      }
    ],
    "max_tokens": 256
  }'

Limitations

• Designed for basic visual understanding tasks
• Not suitable for complex reasoning tasks
• Performance may vary on diverse image types

License

MIT License