Downloads · 30 days
537
9% of all-time downloads
jason-schulz/Gemma-4-26B-A4B-Hermes-VLM-MLX
Gemma-4-26B-A4B-Hermes-VLM-MLX is a image-text-to-text model from jason-schulz. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for mlx. The card lists the license as gemma.
A multimodal Gemma 4 26B-A4B with the Hermes-fine-tuned language tower grafted into Google's original vision stack. Vision and tool calling both work; no retraining was performed on either component.
Downloads · 30 days
537
9% of all-time downloads
All-time downloads
5.8K
Public
Parameters
25.8B
15.4 GB on disk
Likes
11
Public
Click a slice to open those files.
.safetensors15.3 GB · 100%
How the weights are stored.
U3225.2B · 98%
From the Hugging Face model README
A multimodal Gemma 4 26B-A4B with the Hermes-fine-tuned language tower grafted into Google's original vision stack. Vision and tool calling both work; no retraining was performed on either component.
This is the v2 multimodal companion to
jason-schulz/Gemma-4-26B-A4B-Hermes-MLX
(v1, text-only).
This model is a weight graft, not a fine-tune:
Source A (language): the trained Hermes text-only checkpoint —
Gemma-4-26B-A4B-Hermes-MLX — provides the language tower weights,
bit-perfect from the DoRA fine-tune on the Hermes Agent dataset.
Source B (vision): the original google/gemma-4-26b-a4b-it
multimodal release was converted to MLX with matching quantization
(mlx_vlm convert -q --q-bits 4 --q-group-size 64, with a small
patch to mlx_vlm's Gemma 4 quant predicate to align with mlx_lm's
layer-bit choices for shape compatibility).
Splice: every model.X key in the Hermes checkpoint was renamed
to language_model.model.X and substituted into the converted
multimodal checkpoint, replacing the base Gemma 4 language tower
while leaving the vision encoder, projector, and embed_vision
weights untouched.
Validation: the resulting checkpoint loads via mlx_vlm, passes
a text-only tool-calling test (native <|tool_call> syntax intact),
and passes a synthetic vision test (correctly described colors,
shapes, and OCR on a test image).
The graft works because:
| Architecture | Gemma4ForConditionalGeneration (model_type: gemma4) |
| Language tower | Hermes-trained, ~26B params total (~4B active) |
| Vision tower | Original Google Gemma 4 vision encoder + projector |
| MoE | 128 experts, 8 active per token |
| Hidden layers | 30 |
| Hidden size | 2816 |
| Vocab size | 262,144 |
| Context length | 262,144 |
| Quantization | MLX 4-bit affine (group_size=64) with 8-bit router projections; vision tower at 4-bit affine |
| On-disk size | ~14 GB |
| Hardware tested | Apple M5 Max (128 GB) — ~120 tok/s generation, ~15.5 GB peak memory |
Tool calling uses Gemma 4's native format:
<|tool_call>call:function_name{key:<|"|>value<|"|>,...}<tool_call|>
Calls and reasoning content (in the thought channel) are extracted
correctly when served via oMLX
(requires the patches in oMLX's patches/mlx_lm/ and
patches/mlx_vlm/ directories for full Gemma 4 tool-calling support).
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from PIL import Image
model, processor = load("jason-schulz/Gemma-4-26B-A4B-Hermes-VLM-MLX")
image = Image.open("photo.jpg")
messages = [{"role": "user", "content": "What's in this image?"}]
prompt = apply_chat_template(processor, model.config, messages, num_images=1)
output = generate(model, processor, prompt, image=image, max_tokens=200)
print(output.text)
The same model handles text-only conversations and tool calls:
messages = [
{"role": "system", "content": "You have access to tools: write_file, terminal."},
{"role": "user", "content": "Create test.py that prints hello."},
]
prompt = apply_chat_template(processor, model.config, messages, num_images=0)
output = generate(model, processor, prompt, max_tokens=200)
Released under the Gemma Terms of Use. By using this model you agree to Google's Gemma license.