Qwen3.5-4B-MLX-4bit-hybrid

Qwen3.5-4B-MLX-4bit-hybrid

1. Description

This model is a small version of Qwen/Qwen3.5-4B. It operates with MLX on Apple silicon. The model reads text and images. The model has a reasoning mode.

Most of the weights use 4 bits. Some weights use 6 bits or 8 bits. No weight uses 3 bits.

The size of the model is 2660 MB.

2. Why this model is different

Qwen3.5 is a hybrid model. It has two types of layer:

• 8 layers use full attention.
• 24 layers use a gated DeltaNet.

The usual MLX recipe is mixed_3_6. This recipe is for older models. It gives more bits to v_proj, to down_proj and to lm_head. The DeltaNet layers do not have these three parts. Thus the recipe gives 3 bits to all the weights in the 24 DeltaNet layers.

Two of these weights are in_proj_a and in_proj_b. They control the memory of the model. The model calculates the memory decay with this equation:

decay = exp(-exp(A_log) * softplus(a + dt_bias))

The equation contains two exponential functions. Thus a small error in a becomes a large error in the decay. The decay multiplies the memory at each token. Thus the error increases when the text becomes longer.

The two weights are only 3.9 M parameters. This is 0.06 % of the model. But they have a large effect on the quality.

3. The changes in this model

The usual tools do not compress the vision tower. In this model the vision tower uses 4 bits. This change makes the vision tower smaller by 473 MB (from 667 MB to 194 MB). The model uses these bytes for the language layers.

4. Test data

The tests use the bf16 model as the reference. The text tests use the wikitext-2 test set. The tests use 262144 tokens and a sequence length of 2048.

This model is better than the 4-bit model in the quality tests. It is also 374 MB smaller.

The mixed_3_6 model is slow. MLX does not have fast 3-bit code. This model is 2.65 times faster than the mixed_3_6 model.

Long text

The tests measure the KL divergence at each position in a sequence of 8192 tokens. The error increases from the start of the sequence to the end.

5. How to use the model

Install mlx-vlm. Then obey these steps.

1. Download the model.
2. Give the path of the model to the load function.
3. Give a prompt and an image to the generate function.

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("Spakie/Qwen3.5-4B-MLX-4bit-hybrid")
prompt = apply_chat_template(processor, model.config, "What is in this image?", num_images=1)
print(generate(model, processor, prompt, ["image.png"], max_tokens=200))

For text only, use mlx-lm.

from mlx_lm import load, generate

model, tokenizer = load("Spakie/Qwen3.5-4B-MLX-4bit-hybrid")
print(generate(model, tokenizer, "Tell me about Apple silicon.", max_tokens=200))

6. Limits

• The tests for the perplexity and the KL divergence use text only. The tools for these tests do not load the vision tower.
• The tests of the images use three images. These tests do not give a score.
• The tests do not include a standard benchmark set, for example MMLU.
• The model keeps the context length of the initial model.

7. Source

• Initial model: Qwen/Qwen3.5-4B
• License: Apache 2.0
• Format: MLX, affine quantization, mixed bit widths