Qwen3.5-0.8B-MTP-GGUF

Qwen3.5-0.8B-MTP-GGUF

Qwen3.5-0.8B from Alibaba's Qwen team is the smallest model in the Qwen3.5 family, an ultra-compact 0.8B-parameter dense multimodal language model with a hybrid Gated DeltaNet + sparse MoE architecture, 24 layers, 1024 hidden dimension, 248K vocabulary spanning 201 languages, multi-token prediction, and a massive 262K native context window (extensible to 1M+ tokens via YaRN) for unified text and image understanding at extreme efficiency. Designed under the "More Size, Less Waste" philosophy, it achieves 10.5 on the Artificial Analysis Intelligence Index—ranking #383 overall but exceptional for sub-1B models—while running at blazing-fast latencies (0.00s time-to-first-token #12 globally) with ~1.6GB VRAM requirement (BF16) or ~0.5GB in 4-bit quantization, making it ideal for Raspberry Pi, mobile phones, and embedded IoT devices. Apache 2.0-licensed with Ollama/vLLM/llama.cpp support, it excels at lightweight OCR, document parsing, multilingual chatbots, visual QA, and basic coding tasks as the most accessible entry point for on-device multimodal AI without requiring cloud dependencies.

[!NOTE] Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

Quants Usage

(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)

Here is a handy graph by ikawrakow comparing some lower-quality quant types (lower is better):