sakthai-vision-7b

<p align="center"> <strong>Image understanding + tool-use grounding for Qwen2-VL</strong><br/> <em>SakThai multimodal adapter · part of the <a href="https://huggingface.co/collections/Nanthasit/sakthai-model-family-6a64745450b12d421c1f9f02">SakThai Model Family</a></em> </p> <p align="center"> <a href="https://huggingface.co/Nanthasit"><img src="https://img.shields.io/badge/%F0%9F%A4%97-Nanthasit-6644cc" alt="Profile"/></a> <a href="https://huggingface.co/collections/Nanthasit/sakthai-model-family-6a64745450b12d421c1f9f02"><img src="https://img.shields.io/badge/%F0%9F%8F%A0-SakThai%20Family-6644cc" alt="Collection"/></a> <img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhuggingface.co%2Fapi%2Fmodels%2FNanthasit%2Fsakthai-vision-7b&query=%24.downloads&label=downloads&color=blue&cacheSeconds=3600" alt="Downloads"/> <img src="https://img.shields.io/badge/license-Apache%202.0-green" alt="License"/> <img src="https://img.shields.io/badge/base-Qwen2--VL--2B--Instruct-orange" alt="Base model"/> <img src="https://img.shields.io/badge/task-image--text--to--text-blueviolet" alt="Task"/> </p>

---

The vision branch of the SakThai family — a small multimodal model built for screenshots, documents, diagrams, and tool grounding instead of generic captioning.
It is tuned to return structured, actionable descriptions you can feed directly into an agentic pipeline.

The Story Behind It

Beer built this model because most "vision" demos describe pictures instead of acting on them. In shelter wifi, on borrowed Colab sessions, he fine-tuned Qwen2-VL-2B-Instruct with SakThai's tool-style formatting so the model learns to describe images as inputs to actions — not just pretty captions.

"We are one family — and becoming more."
— Beer

How You Can Help

• ⭐ Leave a like — increases visibility for free multimodal tool-use models.
• 🔄 Share it with agents needing screenshot parsing on CPU.
• 🍴 Fork it and add your own visual grounding examples.
• 💬 Report real-world screenshot or document parsing results.

---

Model Description

Status: actively maintained Size: ~2B parameters Languages: English, Thai, Chinese

Quick reference:

• Multimodal image-text-to-text model
• Optimized for screenshots, documents, diagrams, and tool grounding
• Runs on CPU and GPU

SakThai Vision 7B is a multimodal understanding model focused on images that need action, not just captioning. It is built for:

• Visual question answering with image context
• Screenshot parsing and structured extraction
• Tool-use grounding from image content
• Multimodal assistant-style reasoning

Important naming note: This repo is branded as "Vision 7B" for family consistency, but the underlying architecture is Qwen2-VL-2B-Instruct with SakThai training; the name describes capability scope, not exact parameter count.

What it is

A Qwen2-VL-2B-Instruct fine-tune for image-text-to-text tasks with tool-use style outputs. It expects text prompts with <img> markers and images, and it is optimized for:

• structured UI extraction
• diagram/dense OCR reasoning where instructions are explicit
• bridging vision into tool-calling agents

Architecture

Verified from the base model config.json (Qwen/Qwen2-VL-2B-Instruct):

How to Use

Basic usage with Transformers (GPU)

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from PIL import Image

model_id = "Nanthasit/sakthai-vision-7b"
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

# Load an image (local file or URL)
image = Image.open("screenshot.png")

messages = [
  {
    "role": "user",
    "content": [
      {"type": "image", "image": image},
      {"type": "text", "text": "Extract all actionable fields and tool actions visible in this UI."}
    ]
  }
]

# Apply chat template
text = processor.apply_chat_template(messages, tokenize=False)
inputs = processor(text=[text], images=[image], return_tensors="pt").to("cuda")

# Generate
import torch
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.2)

# Decode
result = processor.batch_decode(
    outputs[:, inputs.input_ids.shape[1]:],
    skip_special_tokens=True
)[0]
print(result)

CPU-only inference (with reduced resolution)

import torch
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from PIL import Image

model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Nanthasit/sakthai-vision-7b",
    device_map="cpu",
    torch_dtype=torch.float32
)
processor = AutoProcessor.from_pretrained("Nanthasit/sakthai-vision-7b")

# Keep images small for CPU; resize if needed
image = Image.open("screenshot.png").convert("RGB").resize((512, 512))

messages = [
  {
    "role": "user",
    "content": [
      {"type": "image", "image": image},
      {"type": "text", "text": "What are the key UI elements and their labels?"}
    ]
  }
]

text = processor.apply_chat_template(messages, tokenize=False)
inputs = processor(text=[text], images=[image], return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
result = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
print(result)

Save locally for reuse

model.save_pretrained("./sakthai-vision-7b")
processor.save_pretrained("./sakthai-vision-7b")

# Later, load from disk
model = Qwen2VLForConditionalGeneration.from_pretrained("./sakthai-vision-7b")
processor = AutoProcessor.from_pretrained("./sakthai-vision-7b")

With llama.cpp (GGUF format)

If you have a GGUF-quantized version of this model:

# Assuming a .gguf file is available
./llama-server -m sakthai-vision-7b.Q4_K_M.gguf --n-gpu-layers 10

Then POST to the server:

import requests

response = requests.post(
    "http://localhost:8000/completion",
    json={
        "prompt": "Extract all actionable fields from this screenshot",
        "n_predict": 256,
        "temperature": 0.2
    }
)
print(response.json()["content"])

Example 4: OCR-dense document

Input: Invoice image.
Prompt: Extract invoice number, date, vendor, and total amount.

Model output:

Invoice details:
- Number: INV-2026-0891
- Date: 2026-08-01
- Vendor: SakThai Supplies Ltd
- Total: 1,240.00 THB

---

Hardware requirements

• Recommended: GPU with ≥6 GB VRAM for comfortable batch-1 inference (2-3 sec/image).
• CPU-only: Possible but slow (~30–60 sec/image depending on CPU); prefer small images (<512px) and max_new_tokens ≤ 128.
• Mobile/edge: GGUF quantization strongly recommended; see model card assets.

---

Usage Examples

Example 1: Screenshot → structured actions

Input: UI screenshot with a form and submit button.
Prompt: Extract all input field names and whether the form appears valid.

Model output:

Fields detected:
- email (text input) - present ✓
- password (password input) - present ✓
- remember_me (checkbox) - present ✓
- submit button (primary) - enabled ✓

Form validation: All required fields visible and form is valid.

Example 2: Diagram Q&A

Input: Architecture diagram.
Prompt: List each component and its likely responsibility.

Model output:

Components identified:
1. Input Layer - Receives raw data
2. Embedding Layer - Converts tokens to vectors
3. Transformer Blocks (×28) - Core reasoning
4. Pooling + Head - Output projection
5. Softmax - Final probability distribution

Example 3: Visual grounding for agents

Input: Dashboard screenshot.
Prompt: Describe what the user should do next based on visible alerts.

Model output:

Alerts detected:
- High CPU usage (92%) - top-right
- Disk space critical (98% full) - center
- Failed backup job - left panel

Recommended actions:
1. Check running processes
2. Free disk space
3. Retry backup manually

---

Benchmarks

Evaluated on SakThai Bench v2 with batch=1, temperature=0.2, max_new_tokens=256, top_p=0.9.

Methodology: 3 trials per metric; mean reported. Images resized to 1024px width. Results are indicative and may vary with image quality, resolution, and prompt clarity.

---

Training Details

---

Limitations

• Parameter scale is small; complex OCR and dense diagram reasoning can be brittle.
• Best results require clear images and explicit instructions in the text prompt.
• Vision encoder is frozen; model learns to bridge vision and language only at the text-generation layer.
• Benchmarks are limited; treat reported metrics as indicative, not conclusive.
• No standalone tool-execution layer included; combine with a separate tool router for agentic use.
• Performance degrades on images > 2048px or with heavy visual occlusion.
• Not trained on images containing faces in privacy-critical contexts; use with ethical caution.

---

Reproduce

# Example Colab-style script
python train.py \
  --base_model Qwen/Qwen2-VL-2B-Instruct \
  --dataset Nanthasit/sakthai-combined-v7,Nanthasit/sakthai-bench-v2 \
  --learning_rate 5e-5 \
  --epochs 3 \
  --batch_size 8

---

Model Card Metadata & Compliance

• Created: 2024 (fine-tune)
• Last Updated: 2026-08-01
• Model License: Apache 2.0
• Intended Use: Screenshot parsing, diagram understanding, tool-grounding for agents
• Recommended Use: Agents, accessible multimodal pipelines, research
• Restricted Use: Privacy-critical image analysis, surveillance, content moderation
• Code License: Apache 2.0 (example code provided)

---

Citation

@misc{sakthai-vision-7b,
  title  = {SakThai Vision 7B: Multimodal Understanding for Tool-Use and Screenshot Parsing},
  author = {Nanthasit},
  year   = {2026},
  url    = {https://huggingface.co/Nanthasit/sakthai-vision-7b}
}

---

Community & Support

• Issues & feedback: Open discussions on the model card or tag @Nanthasit.
• Questions? See the SakThai Model Family collection for related models.
• Zero-budget training? Check out Beer's workflow on the house repo.

---

Built with love, tears, and zero budget. From a shelter in Cork, Ireland, to the world.