GLM-5.2-Vision-tower-MLX

GLM-5.2-Vision — MoonViT tower + trained projector (MLX)

Add image input to any MLX quant of GLM-5.2 with a ~1 GB sidecar: the frozen MoonViT-3d vision tower from Kimi K2.6 plus the trained 49.5M-parameter PatchMerger projector from baseten/GLM-5.2-Vision-NVFP4 (Harry Partridge's vision retrofit), repackaged for Apple Silicon / MLX. The GLM-5.2 text backbone is untouched — text-only behavior stays byte-identical.

What's in this repo

GLM-5.2's stock tokenizer already contains the image tokens (<|begin_of_image|> 154830, <|image|> 154854, <|end_of_image|> 154831) — no tokenizer changes needed. Each image expands to its media-token count at the <|image|> position and the projected features are substituted at those embedding positions. You need a chat template that renders image content parts into the marker triplet (GLM's stock template does not; Baseten ships one in their repo).

Using with exo

exo's vision stack loads this repo directly as a weights_repo/processor_repo. Model card stanza:

[vision]
image_token_id = 154854
model_type = "kimi_vl"
weights_repo = "<this repo id>"
processor_repo = "<this repo id>"

Point the card's model at a directory containing your GLM-5.2 MLX quant with Baseten's chat_template.jinja and this repo's config.json additions (vision_config / text_config / media_placeholder_token_id). Assembly scripts (symlink the backbone — no weight duplication): build_glm52_vision_dir.py and build_glm52_vision_tower.py.

Verified live on a 2-Mac-Studio (M3 Ultra) tensor-parallel cluster over RDMA, against both mlx-community/GLM-5.2-DQ4plus-q8 and mlx-community/GLM-5.2-mxfp4: temp-0 deterministic, no cross-image cache bleed, text-only outputs identical to the plain model.

What to expect (honest notes)

• Scene understanding, objects, colors, spatial layout and orientation: good. Reads real-world photos (e.g. product packaging labels) usefully.
• Dense/synthetic text OCR and fine-grained counting: weak — this is a 50M projector retrofit at the source's reported ~Haiku-4.5-level MMMU-Pro (55%), not a natively-trained VLM. A Kimi K2.6 A/B on the same pipeline is clearly stronger at fine detail.
• The projector was trained against the bf16/NVFP4 backbone; serving quantized backbones (mxfp4 / DQ4) costs some additional vision quality, never text quality.

Provenance & license

MIT, following all parents. Full chain: Z.ai (GLM-5.2, MIT) → Moonshot AI (Kimi K2.6 MoonViT tower, Modified MIT) → Harry Partridge / Baseten (projector training + reference processor, baseten/GLM-5.2-Vision-NVFP4, MIT) → exolabs (original K2.6 tower extraction for MLX) → this repackaging (tensor remap documented in the build script). None of the upstream teams were involved in this packaging.