Qwen3-VL-8B-Instruct

<div align="center">

👁️‍🗨️ Qwen3-VL-8B-Instruct — GGUF Edition

An AI that can read, see, and think — now shrunk down to run on your own machine.

</div> <br/>

[!TIP] In a hurry? Jump to 🧊 Quantizations & Hardware to pick a version that fits your machine, or 🚀 Quick Start to start chatting.

---

🤔 What am I looking at?

Think of Qwen3-VL as a language model that also has eyes. You can show it a photo, a screenshot, a chart, a scanned document, or even a video, and talk to it about what it sees — the same way you'd chat with a normal AI assistant, just with pictures added to the conversation.

This particular page is for Qwen3-VL-8B-Instruct — a mid-sized version of the model that's small enough to run on a good laptop or desktop, but still genuinely capable.

🏭 What's happening in this repository?

[!NOTE] This is a Talisma AI Studio conversion repository. We take the original model — Qwen/Qwen3-VL-8B-Instruct — and:

1. 🔄 Convert it from its original format into GGUF
2. 🗜️ Quantize it into several smaller sizes
3. 👁️ Extract and package the vision encoder (mmproj) separately
4. ☁️ Upload everything here for the team to download and run locally

GGUF + quantization is what makes it possible to run an 8-billion-parameter vision model on ordinary hardware instead of a server rack.

---

🌟 Why this model is a big deal

Qwen's team describes Qwen3-VL as the most capable vision-language model in the Qwen family so far. In plain terms, here's what changed:

<table> <tr> <td width="50%" valign="top">

🤖 It can operate apps for you Recognizes buttons, menus, and UI elements on a screen well enough to click through tasks on its own — a real step toward "AI that uses your computer."

🎨 It can turn pictures into code Show it a screenshot or a hand-drawn diagram, and it can generate the matching Draw.io, HTML, CSS, or JS.

📐 It understands where things are Judges object position, viewpoint, and occlusion — not just "what" is in the image, but "where" and "how it's arranged," including in 3D.

</td> <td width="50%" valign="top">

📚 It has a very long memory Natively handles 256K tokens of context (expandable to 1M) — enough to recall details across an entire book or an hours-long video.

🧮 It's better at STEM reasoning Stronger at math and science problems that require step-by-step, evidence-based reasoning.

🔤 It reads more languages, more reliably OCR now covers 32 languages (up from 19), and holds up better in blur, tilt, low light, and messy handwriting.

</td> </tr> </table> <details> <summary>🧠 <b>Curious about the architecture? (click to expand)</b></summary> <br/>

Three technical upgrades power the improvements above:

</details>

---

🧬 Model Specifications

---

📊 How good is it, really?

Qwen benchmarked Qwen3-VL-8B-Instruct across a wide range of categories — both multimodal (image/video + text) and pure text tasks — comparing it against the smaller Qwen3-VL-4B and other models in its class.

<details> <summary>📈 <b>What gets tested (click to expand)</b></summary> <br/>

Multimodal categories: STEM & puzzles, general visual Q&A, subjective/instruction-following, OCR & document/chart understanding, 2D/3D grounding, multi-image reasoning, embodied & spatial understanding, video understanding, agentic (GUI) tasks, and fine-grained perception.

Pure text categories: general knowledge, reasoning, coding, instruction following, subjective evaluation, agentic tool use, and multilingual tasks.

Across both sets, the 8B Instruct model consistently scores higher than the smaller 4B version, and holds its own against other models in its size class — particularly in OCR, document understanding, and agentic/GUI tasks, which were a major focus of this release.

For the full, exact benchmark numbers (they're extensive), see the official model card.

</details>

---

🧊 Quantizations & Hardware

[!IMPORTANT] Status: conversion in progress. The table below reflects our planned quantization lineup and typical size/memory expectations for an 8B dense model — not yet Talisma-measured numbers. Once files are uploaded, this section will be updated with real file sizes and confirmed hardware guidance.

The vision encoder (mmproj) is packaged and quantized separately from the language model — commonly at Q8_0 or F16 — since it's much smaller and keeping it higher-precision preserves visual accuracy even when the language model itself is more heavily compressed.

<div align="center">

Smaller & Faster  ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  Bigger & Sharper
                  Q2_K   Q3_K_M   Q4_K_M   Q5_K_M   Q6_K   Q8_0   F16

Most people should start with Q4_K_M or Q5_K_M.

</div> <details> <summary>💻 <b>Not sure what quant/RAM means? (click to expand)</b></summary> <br/>

Think of quantization like saving a photo at different compression levels. A Q4 file is like a smaller JPEG — noticeably smaller, and almost as good to the eye. A F16 file is like the original, uncompressed photo — perfect quality, but heavy to store and slow to load. Most people can't tell the difference between Q5 and full quality in everyday use, but the file size difference is huge.

Rule of thumb: your available RAM should comfortably exceed the model file size — actual usage also depends on context length, image size, and how many things you're running at once.

</details>

---

🚀 Quick Start

<details open> <summary><b>🐍 Python (Transformers)</b> — for the original model, useful for testing/reference</summary>

pip install git+https://github.com/huggingface/transformers

from transformers import Qwen3VLForConditionalGeneration, AutoProcessor

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-VL-8B-Instruct", dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=128)
trimmed = [out[len(inp):] for inp, out in zip(inputs.input_ids, generated_ids)]
print(processor.batch_decode(trimmed, skip_special_tokens=True))

💡 For multi-image or video inputs, enabling flash_attention_2 is recommended for speed and memory savings.

</details> <details> <summary><b>🦙 llama.cpp</b> — once GGUF files are uploaded</summary>

llama-mtmd-cli \
  -m Qwen3VL-8B-Instruct-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3VL-8B-Instruct-F16.gguf \
  --image photo.jpg \
  -p "What's going on in this picture?"

</details> <details> <summary><b>💻 LM Studio / Open WebUI</b></summary>

Once uploaded here, download the language model file and its matching mmproj file, load them together, and chat as normal — attach an image whenever you like.

</details>

---

🎛️ Generation Settings

Different starting points depending on whether you're mostly using images/video, or pure text:

(greedy=false for both — the model samples rather than always picking the single most likely next word.)

---

🎯 What can I actually use this for?

No jargon — just everyday examples of what this model is good at:

---

🚧 Things to keep in mind

[!WARNING]

• It can be confidently wrong — always double-check important facts, especially in specialized domains.
• Quantized versions trade a little quality for size — the more compressed the file, the more small errors can creep in.
• Video and multi-image inputs need more memory and time than a single image or plain text.
• Not a substitute for professional judgment — avoid relying on it alone for medical, legal, financial, or safety-critical decisions.

---

🗂️ What's in this repository

---

📚 References & Citation

• Original Model — Qwen/Qwen3-VL-8B-Instruct
• Qwen3 Technical Report
• Qwen2.5-VL Technical Report
• Qwen2-VL Paper
• Qwen-VL Paper
• llama.cpp

<details> <summary>📖 <b>BibTeX citation</b></summary>

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report},
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388},
}

</details>

---

<div align="center">

License: Apache 2.0 · Original model by the Qwen Team · GGUF conversion & quantization by Talisma AI Studio

Built for the Talisma AI Studio Model Registry 🧠

</div>