Qwen-3.5-4B-Conversational

<div align="center"> <img src="https://huggingface.co/front/assets/huggingface_logo-noborder.svg" width="60"/>

🐉 Qwen‑3.5‑4B‑Conversational

A thinking model that fits in your pocket — now in GGUF.

📄 Base Model Card · 📘 Qwen3.5 Blog · 💬 Talisma AI Studio

</div> <br/> <div align="center">

Qwen3.5‑4B outscores the 30B‑parameter Qwen3‑30B‑A3B‑Thinking‑2507 on agentic tool‑use — at roughly 1/7th the size.

</div>

---

🍿 TL;DR — what even is this? (for the non‑techie reader)

Qwen3.5‑4B is a small, sharp "thinking" AI model built by the Qwen Team — it reads text, images and even video, reasons step‑by‑step before answering, and speaks 201 languages. Talisma AI Studio took the original model and shrank it down using a technique called quantization, exporting it as several GGUF files of different sizes — the same brain, packaged at different levels of compression.

Think of it like saving the same photo as a tiny thumbnail, a normal JPEG, or a full‑resolution RAW file — same picture, different file sizes for different needs. Smaller file = fits on more modest hardware and runs faster; larger file = closer to the original model's full sharpness. Pick the one that matches your laptop, server, or phone.

This checkpoint keeps Qwen3.5's thinking‑by‑default behaviour, its vision understanding, and its native 262K‑token context (extendable to 1M tokens) — just packaged to actually run on the hardware you already own.

---

✨ Why this model is actually exciting

<table> <tr> <td width="33%" valign="top">

🧠 Thinks before it speaks

Generates internal <think>...</think> reasoning before the final answer — strong on math, coding, and multi‑step logic.

</td> <td width="33%" valign="top">

👀 Sees text, images & video

Native multimodal training (not bolted‑on) — reads documents, screenshots, charts, photos, and can summarize video clips.

</td> <td width="33%" valign="top">

🪶 Punches way above its weight

On agentic tool‑use (TAU2‑Bench) this 4B model beats the 30B‑parameter Qwen3‑30B‑A3B‑Thinking‑2507 — see the bar above.

</td> </tr> <tr> <td width="33%" valign="top">

🌍 Speaks 201 languages

Broad multilingual coverage with strong scores on MMMLU, MMLU‑ProX, and WMT24++ translation benchmarks.

</td> <td width="33%" valign="top">

🛠️ Built for agents

Native tool/function‑calling, strong BFCL‑V4 and TAU2‑Bench scores — plugs into agent frameworks and automations.

</td> <td width="33%" valign="top">

📚 Remembers a lot

262,144‑token native context, extensible to 1,010,000 tokens via YaRN — feed it entire codebases or long transcripts.

</td> </tr> </table>

---

🎯 Who is this for & what can you build with it?

<details> <summary><b>🖱️ Click to expand real‑world use cases</b></summary> <br/> </details>

---

🔬 Under the hood — model specifications

Extracted from the official Qwen/Qwen3.5-4B model card.

<details> <summary>📎 Architecture notes (click to expand)</summary> <br/>

Qwen3.5 uses a hybrid architecture repeated 8 times, where each block is 3 layers of Gated DeltaNet (a linear‑attention variant) followed by FFN, then 1 layer of Gated Attention (full attention) followed by FFN:

8 × ( 3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN) )

• Gated DeltaNet: 32 linear‑attention heads for V, 16 for Q/K, head dimension 128 — cheap, fast, and good at long sequences.
• Gated Attention: 16 heads for Q, 4 for KV, head dimension 256, with rotary position embeddings (dim 64) — brings back full‑sequence awareness periodically so nothing gets lost over long context.
• Multi‑Token Prediction (MTP): trained to predict several tokens ahead, enabling speculative decoding for faster inference.

This mix gives Qwen3.5 the speed of a linear‑attention model with the long‑range awareness of full attention — a big part of why the 4B model competes with much larger dense/thinking models on agentic and long‑context tasks.

</details>

---

📊 Benchmark performance (Qwen3.5‑4B, base model scores)

<div align="center">

Agentic tool‑use — small model, big result

_{TAU2‑Bench (avg. of 3 domains) — higher is better}

</div> <br/> <details> <summary><b>📈 Full language benchmark table (click to expand)</b></summary> <br/>

Empty cells (--) indicate scores not yet available or not applicable. Full table with all 40+ benchmarks is on the base model card.

</details> <details> <summary><b>🖼️ Vision‑language benchmark table (click to expand)</b></summary> <br/>

Full vision‑language table on the base model card.

</details>

---

🧊 Why we quantized this model

Full‑precision Qwen3.5‑4B ships as ~9 GB of BF16 weights — great on a workstation GPU, painful on a laptop or edge box. Quantization compresses those weights (from 16‑bit down to as low as 2‑bit) using calibrated, block‑wise scaling ("K‑quants"), trading a small amount of precision for large practical wins:

The trade‑off: very aggressive quantization (Q2_K) can soften factual precision and coding accuracy slightly. K‑quants (the _K_M and _K variants below) mitigate this with smarter, non‑uniform bit allocation, so quality holds up much better than naive quantization at the same file size.

---

📦 Files in this repository

<div align="center">

_{Relative file sizes of each quantization (main language‑model weights)}

</div> <br/>

💡 Rule of thumb: allocate roughly file size + 2 GB of RAM/VRAM for comfortable inference (extra headroom for context and KV‑cache at longer sequence lengths). Add another ~1 GB if you're also loading the mmproj vision projector.

<details> <summary>🤔 Not sure which one to pick? Click here</summary> <br/>

• Just exploring / laptop with 4–8 GB RAM → go with Q4_K_M
• Have a decent GPU (6 GB+ VRAM) and want the best answers → go with Q6_K or Q8_0
• Fine‑tuning, research, or need bit‑for‑bit fidelity → use BF16
• Running on something really small (SBCs, older phones) → Q2_K or Q3_K_M, but expect softer reasoning quality
• Need image or video understanding → always download mmproj-Qwen3.5-4B-BF16.gguf alongside your chosen quant

</details>

---

🚀 Quick start

<details open> <summary><b>🦙 Using llama.cpp</b></summary>

# Download your preferred quantization + the vision projector
huggingface-cli download talisma/Qwen-3.5-4B-Conversational \
  Qwen3.5-4B-Q4_K_M.gguf mmproj-Qwen3.5-4B-BF16.gguf --local-dir .

# Text chat
./llama-cli -m Qwen3.5-4B-Q4_K_M.gguf -p "Explain photosynthesis to a 10 year old." -cnv

# With an image
./llama-cli -m Qwen3.5-4B-Q4_K_M.gguf --mmproj mmproj-Qwen3.5-4B-BF16.gguf \
  -p "Describe this image." --image photo.jpg

</details> <details> <summary><b>💻 Using Ollama</b></summary>

ollama run hf.co/talisma/Qwen-3.5-4B-Conversational:Q4_K_M

</details> <details> <summary><b>🐍 Using Python (transformers, base model reference)</b></summary>

from transformers import AutoProcessor, AutoModelForCausalLM

MODEL_ID = "Qwen/Qwen3.5-4B"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, dtype="auto", device_map="auto"
)

messages = [{"role": "user", "content": "Type \"I love Qwen3.5\" backwards"}]

inputs = processor.apply_chat_template(
    messages, tokenize=True, return_dict=True, return_tensors="pt",
    add_generation_prompt=True
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=2048)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

</details>

---

⚙️ Recommended settings

• Qwen3.5 thinks by default, generating <think>...</think> content before its final answer. Disable it via enable_thinking: False in your chat template if you just want direct responses.
• Recommended output length: 32,768 tokens for most queries; up to 81,920 tokens for hard math/coding benchmarking.
• For long‑context work beyond 262K tokens, enable YaRN RoPE scaling (see the base model card for the exact config) — but keep at least 128K tokens of context to preserve full thinking capability.

<blockquote> 🧵 <b>Multi‑turn tip:</b> in conversation history, keep only the model's <i>final</i> answers — strip out prior "thinking" content before the next user turn. This is handled automatically by the provided chat template. </blockquote>

---

🧬 Provenance & attribution

Base model:      Qwen/Qwen3.5-4B        (Qwen Team, Alibaba)
Quantized by:    Talisma AI Studio
Format:          GGUF
Method:          llama.cpp quantization
License:         Apache 2.0 (inherited from base model)

Talisma AI Studio evaluates and benchmarks open‑weight models (Qwen, Gemma, Llama, Mistral, Phi, and embedding models) to power Talisma's enterprise CXM/CRM and Agentic AI products. This repository packages the Qwen Team's Qwen3.5‑4B for efficient, on‑premise, and edge deployment.

---

⚠️ Good to know

<details> <summary>Limitations & responsible use</summary> <br/>

• Like all LLMs, this model can produce incorrect or outdated factual statements — verify important outputs.
• Lower‑bit quantizations (especially Q2_K) can show a noticeable drop in coding and precise‑math accuracy compared to BF16.
• Video input is processed as sampled frames (default 2 fps); very long videos should use a lower frame rate.
• Refer to the Qwen Team's official usage guidance and Apache 2.0 license terms for deployment considerations.

</details>

---

📖 Citation

@misc{qwen3.5,
    title  = {{Qwen3.5}: Towards Native Multimodal Agents},
    author = {{Qwen Team}},
    month  = {February},
    year   = {2026},
    url    = {https://qwen.ai/blog?id=qwen3.5}
}

<div align="center">

---

Made with 💜 by Talisma AI Studio · Base model by Qwen Team (Alibaba)

🤗 Talisma on Hugging Face · 📄 License: Apache 2.0

</div>