KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit

sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit

A vision-language model created by grafting the vision tower of /unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit onto the language model of mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit.

The finetuned model was language-only (its vision tower had been stripped during training), but was the best model for agentic coding i ever tested for 32gb systems (very good multi turn tool calling and small thinking blocks for a qwen variant). This merge fully restores multimodal capability by re-attaching the original vision encoder and projector from the base VLM, while keeping the finetuned language weights intact.

• Architecture: Qwen3_5MoeForConditionalGeneration (qwen3_5_moe)
• Language model: from mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit (finetuned, quantized)
• Vision tower: from unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit (unmodified, bf16)
• Format: MLX
• Runs with: mlx-vlm (https://github.com/Blaizzy/mlx-vlm), oMLX prefered

Verify image capability

Install:

pip install mlx-vlm

Command line:

python -m mlx_vlm.generate \
  --model YOUR_MODEL_NAME \
  --image path/to/image.jpg \
  --prompt "Describe this image. Extract all text" \
  --max-tokens 5000

Python:

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("YOUR_MODEL_NAME")
config = model.config

prompt = apply_chat_template(
    processor, config,
    "Describe this image. Extract all text",
    num_images=1,
)
output = generate(model, processor, prompt, ["path/to/image.jpg"],
                  max_tokens=5000, verbose=True)
print(output)

How the merge was done

The two source models share the same qwen3_5_moe architecture and an identical namespace convention, which made a clean weight-level merge possible without any retraining, fine-tuning, or projector re-alignment.

1. Namespace split

All tensors in both models fall into two disjoint prefixes:

• language_model.* -> Language model + LM head -> taken from the FINETUNE
• vision_tower.* -> Vision encoder + merger -> taken from the BASE VLM

The merge is a straightforward union: every language_model.* tensor from the finetune, and every vision_tower.* tensor from the base VLM.

2. Dimension compatibility

The vision projector (vision_tower.merger) outputs vision_config.out_hidden_size = 2048, which matches the language model's text_config.hidden_size = 2048. Vision features therefore project directly into the LM embedding space with no adapter needed. Image features are injected at the image_token_id position (single-point injection; deepstack_visual_indexes is empty).

3. Precision and quantization

• Language model: retains the finetune's mixed quantization scheme -- 4-bit switch_mlp experts, 8-bit attention / shared experts / embeddings / LM head (group size 64, affine). The finetune's quantization map was carried over verbatim.
• Vision tower: kept at full precision (bfloat16), exactly as shipped in the base VLM. MLX supports this mixed quantized-LM / bf16-vision setup.

4. Configuration

The merged config.json uses the finetune's config as the base (it holds the correct quantization map and text_config), with the following vision-related fields grafted in from the base VLM:

• vision_config
• image_token_id, video_token_id
• vision_start_token_id, vision_end_token_id

The multi-token-prediction head was not carried over (mtp_num_hidden_layers remains 0, matching the finetune). The image preprocessor (preprocessor_config.json) were taken from the base VLM, since the finetune's tokenizer configuration was text-only. The tokenizer vocabulary is identical between both sources (vocab_size = 248320, same special-token IDs), so the base VLM's template is fully compatible.

5. Verification

The merged weights were verified byte-for-byte via SHA-256 hashes of each tensor, comparing the merged output against both sources:

• All 1757 language_model.* tensors are byte-identical to the finetune.
• All 333 vision_tower.* tensors are byte-identical to the base VLM.

This model is a derivative combining:

• Language model: mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit
• Vision tower: unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit

License: apache-2.0. You must comply with the licenses of BOTH source models. Please cite and credit the original authors of both the base VLM and the finetune.

Reproduction

The merge was performed with a script (scripts/merge.py) that:

1. Loads all safetensors from both source directories natively in MLX (preserving bfloat16 and packed-quant dtypes).
2. Selects language_model.* tensors from the finetune and vision_tower.* tensors from the base VLM.
3. Builds the merged config (finetune config + base vision fields).
4. Copies the image preprocessor config from the base VLM, rest from finetune.
5. Saves the merged safetensors with a model.safetensors.index.json.

The merge was split into Hugging Face splittensors using the scripts/split.py.

The model was verified for correct tensors (checking text tensors were untouched) using the scripts/verify.py.

You should be able to do your own merges with the script as long as the two models are compatible (check for same dimensions, same tokenizer size). Verify and check vision capabilities (f.e. ocr a text).

The model uses unsloth chat_template with default reasoning_preserve=true. If you prefer the original chat_template replace chat_template.json with chat_templat.json.org or any you like.

Recommended settings (coding/thinking):

Though finetuned, this is still Qwen3.6, so stick close to the official Qwen recommendations (temp 0.7, top_p=0.95, min_p=0.0, top_k=20, repetition_penalty=1.0, presence_penalty=0.0). Avoid temp < 0.4, as this can cause the model to loop!!!

My personal recommendation (forced! in oMLX when required, as f.e. copilot likes to use temp 0.0): temp 0.6, top_p=0.85, min_p=0.0, top_k=20, repetition_penalty=1.0, presence_penalty=0.0, thinking_budget=none, enable_thinking=true, preserve_thinking=true, output_token_limit=16384