Qwen3.5-9B-AltText-v4-GGUF

Qwen3.5-9B-AltText-v4 - GGUF

This repository contains GGUF quantized models for Jakelolipopp/Qwen3.5-9B-AltText-v4-LORA, fine-tuned from base model unsloth/Qwen3.5-9B. Full merged safetensors weights are available at Jakelolipopp/Qwen3.5-9B-AltText-v4-merged.

The models and multimodal vision projector (mmproj) were converted and quantized using llama.cpp.

Available Quantization Files

---

Quantization Details

Base Models

• Q8_0: Near-lossless quality, highest precision (recommended for maximum accuracy).
• Q6_K: High quality, virtually indistinguishable from full precision with noticeable size reduction.
• Q5_K_M: Great balance between quality and speed/VRAM usage (Recommended default).
• Q4_K_M: Fast inference, compact size, great for lower memory devices.
• Q3_K_M / Q2_K: Ultra-compressed for minimal resource footprints.

Multimodal Projectors (mmproj)

• mmproj-...-BF16.gguf: Full bfloat16 vision projector for maximum visual fidelity.
• mmproj-...-Q8_0.gguf: 8-bit quantized vision projector with lower VRAM overhead.

---

How to Run with llama.cpp

Make sure you pass both the model (-m) and the vision projector (--mmproj):

# Run interactive CLI with an image
./llama-cli \
  -m Qwen3.5-9B-AltText-v4-Q5_K_M.gguf \
  --mmproj mmproj-Qwen3.5-9B-AltText-v4-BF16.gguf \
  --image /path/to/image.jpg \
  -p "Generate detailed alternative text describing this image:" \
  -c 4096 --temp 0.2

Running Server Mode

./llama-server \
  -m Qwen3.5-9B-AltText-v4-Q5_K_M.gguf \
  --mmproj mmproj-Qwen3.5-9B-AltText-v4-BF16.gguf \
  --port 8080 -c 4096

---

How to Run with Ollama

Create a Modelfile:

FROM ./Qwen3.5-9B-AltText-v4-Q5_K_M.gguf
MMPROJ ./mmproj-Qwen3.5-9B-AltText-v4-BF16.gguf

PARAMETER temperature 0.2
PARAMETER top_p 0.95

Build and run the model:

ollama create qwen3.5-9b-alttext-v4 -f Modelfile
ollama run qwen3.5-9b-alttext-v4 "Describe this image: /path/to/image.png"