GLM-5.2-Vision-mmproj-GGUF

GLM-5.2 Vision mmproj for llama.cpp

This repository makes the existing unsloth/GLM-5.2-GGUF UD-Q3_K_XL text model accept image input through llama.cpp.

It does not modify, retrain, or include the 3-bit text weights. Vision is attached at runtime through a separate F16 GGUF containing the MoonViT vision tower and trained PatchMerger projector from baseten/GLM-5.2-Vision-NVFP4.

This is an unofficial conversion and llama.cpp integration, not a newly trained checkpoint. Baseten, Z.ai, Moonshot AI, Unsloth, and the llama.cpp project were not involved in this conversion.

Files

The text GGUF is intentionally not duplicated here. Download UD-Q3_K_XL/GLM-5.2-UD-Q3_K_XL-00001-of-00009.gguf and its split shards from Unsloth.

Compatibility

The patch was tested against llama.cpp commit:

ebd048fc5e4b43ec4e0b4abe0b9bf66e1724dad0

Stock llama.cpp at that commit does not recognize PROJECTOR_TYPE_GLM5V. Apply the included patch before building:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout ebd048fc5e4b43ec4e0b4abe0b9bf66e1724dad0
git apply /path/to/glm5v-mmproj-vision.patch

cmake -B build-cuda -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-cuda --target llama-server -j

The patch adds 44 lines across six files. It reuses llama.cpp's existing Kimi-K2.5 MoonViT graph, but assigns a separate projector type so mtmd emits GLM's image delimiters:

<|begin_of_image|>
<|end_of_image|>

Kimi's <|media_begin|> and <|media_end|> tokens are not present in the GLM-5.2 vocabulary.

Serving

./build-cuda/bin/llama-server \
  --model /path/to/GLM-5.2-UD-Q3_K_XL-00001-of-00009.gguf \
  --mmproj /path/to/mmproj-GLM-5.2-Vision-f16.gguf \
  --alias glm-5.2 \
  --host 0.0.0.0 \
  --port 8006 \
  --n-gpu-layers 999 \
  --ctx-size 524288 \
  --parallel 1 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --flash-attn on \
  --jinja

The tested deployment used 8x RTX 6000 Ada GPUs, a 524,288-token server context, one slot, and the Unsloth UD-Q3_K_XL text GGUF.

Example OpenAI-compatible request:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8006/v1", api_key="local")
response = client.chat.completions.create(
    model="glm-5.2",
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://example.com/screenshot.png"
                },
            },
            {"type": "text", "text": "What is wrong with this screenshot?"},
        ],
    }],
    temperature=0,
    max_tokens=1024,
)
print(response.choices[0].message.content)

Evaluation

The included exploratory evaluation uses four programmatically generated image cases and programmatic ground truth; it does not use a judge model. Pi and Kilo were each repeated three times.

One Kilo code run returned empty at a 200-second timeout and is counted as zero. The stored server evidence does not establish whether that event was a client, harness, model, or interaction failure, so no narrower attribution is made here.

The labelled-chart diagnostic returned all five printed values exactly. The unlabelled-axis case remained weak, suggesting that magnitude estimation from bar height/gridlines is more difficult than OCR and label/value binding in this small test. This is an exploratory four-case suite, not a general vision benchmark.

Raw artifacts are preserved under eval/. Note that the programmatic OCR score checks required identifiers and bug terms; the exact-transcription claim in the example above was separately verified from the stored response.

Limitations

• Images only. Video is unsupported. The source checkpoint does not ship the temporal position embedding needed for correct video inference.
• This requires the included out-of-tree llama.cpp patch until equivalent support is available upstream.
• The projector was not retrained for the Q3 text backbone.
• The official Baseten reference uses NVFP4 text weights and Blackwell GPUs. No controlled BF16-vs-Q3 or NVFP4-vs-Q3 vision-quality comparison was performed. The included evaluation cannot establish quantization damage.
• Fine-grained counting and axis-based magnitude estimation can be unreliable.
• The provided context and GPU settings are the tested configuration, not minimum hardware requirements.

Provenance

The converted source components came from baseten/GLM-5.2-Vision-NVFP4 revision:

f6eab6117386a0c69152fdf272dc65bfd0254f9f

That release states that the text backbone and MoonViT tower are frozen and that only the 49.5M-parameter PatchMerger projector was trained. This repository contains no GLM text weights.

Related upstream sources:

• GLM-5.2 text model: zai-org/GLM-5.2
• Q3 text GGUF: unsloth/GLM-5.2-GGUF
• Vision release: baseten/GLM-5.2-Vision-NVFP4
• MoonViT source model: moonshotai/Kimi-K2.6
• Inference runtime: ggml-org/llama.cpp

License

This bundle carries mixed upstream terms:

• Baseten states that the trained projector is released under the MIT License.
• The redistributed MoonViT tower comes from Kimi-K2.6 under the Modified MIT License.
• The included llama.cpp patch is distributed under llama.cpp's MIT terms.

Kimi-K2.6's Modified MIT License requires products or services with more than 100 million monthly active users, or more than USD 20 million in monthly revenue, to prominently display Kimi K2.6 in their user interface.

See LICENSES/ and THIRD_PARTY_NOTICES.md. Redistributed upstream weights remain subject to their original terms. This summary is practical guidance, not legal advice.