vision-27b-q4-gguf

herberttung/vision-27b-q4-gguf

Vision 27B (Q4_K_M, GGUF) — 4-bit GGUF quantization of a 27B vision-language model (image + text input, text output) for use with llama.cpp, LM Studio, and Ollama.

Files

Both files must be in the same folder when loading.

Quick start (llama.cpp)

llama-server -m vision-27b-q4-km.gguf --mmproj mmproj-f16.gguf -c 65536 --flash-attn on -ngl 99 --jinja --port 8080

This exposes an OpenAI-compatible API at http://127.0.0.1:8080/v1.

• Current llama.cpp builds require --flash-attn on, off, or auto. A bare --flash-attn causes the next argument to be parsed as its value.
• --jinja enables the model's Jinja chat template and OpenAI-compatible tool calling for OpenCode.
• Increase -c up to 262144 if you have the memory (the model supports it).
• -ngl 99 offloads all layers to GPU; lower it if you run out of VRAM.
• Add --top-k 20 to match the model's sampling defaults.

OpenCode (AI coding agent) settings

OpenCode works with this model over any OpenAI-compatible server (llama-server, LM Studio, Ollama).

1. Serve the model (Quick start above), or load it in LM Studio / Ollama.

2. Add the provider to opencode.json (project) or ~/.config/opencode/opencode.json (global):

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "vision-local": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Vision 27B (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "vision-27b-q4-km": {
          "name": "Vision 27B Q4_K_M",
          "limit": {
            "context": 65536,
            "output": 8192
          },
          "options": {
            "temperature": 0.7,
            "topP": 0.95
          }
        }
      }
    }
  },
  "model": "vision-local/vision-27b-q4-km"
}

3. Start opencode, run /models, and pick Vision 27B Q4_K_M.

Optimum settings

Sampling notes: the model's generation config defaults to temperature 1.0, top_p 0.95, top_k 20. For agentic coding in OpenCode, 0.2-0.7 temperature is more reliable (fewer tool-call loops, more deterministic edits).

Image input: OpenCode sends image attachments automatically. The mmproj-f16.gguf file enables the vision encoder — keep both files together.

Other servers: LM Studio uses http://127.0.0.1:1234/v1, Ollama uses http://127.0.0.1:11434/v1 — swap the baseURL accordingly.

License

Apache-2.0.