Mistral-Small-4-119B-2603-MLX-8bit

Mistral Small 4 119B — MLX 8-bit

8-bit quantized MLX version of mistralai/Mistral-Small-4-119B-2603 for inference on Apple Silicon.

Model Details

Requirements

• Apple Silicon Mac with 128GB+ unified memory
• Python 3.10+
• mlx-lm with Mistral4 architecture support

Important: mlx-lm compatibility

As of 2026-03-21, the stable mlx-lm pip release does not support model_type: mistral4. You need mlx-lm from the main branch or a patched version.

# Option 1: Install from main (once Mistral4 support is merged)
pip install git+https://github.com/ml-explore/mlx-lm.git

# Option 2: Install stable + manually add mistral4.py
pip install mlx-lm
# Then add mistral4.py to mlx_lm/models/ and update mistral3.py

Usage

from mlx_lm import load, generate

model, tokenizer = load("sachin-sith/Mistral-Small-4-119B-2603-MLX-8bit")

messages = [{"role": "user", "content": "Explain quantum computing in simple terms."}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
response = generate(model, tokenizer, prompt=text, max_tokens=256)
print(response)

Conversion Details

Mistral Small 4 uses a novel architecture not yet supported in the stable mlx-lm release. A custom mistral4.py model implementation was created, handling:

• MLA (Multi-head Latent Attention): Compressed KV cache via latent projections (q_lora_rank=1024, kv_lora_rank=256, qk_nope_head_dim=64, qk_rope_head_dim=64)
• MoE with shared experts: 128 routed experts + 1 shared expert, top-4 routing with softmax gate
• FP8 dequantization: Original weights are FP8 with per-tensor scalar scale factors
• Fused expert weight splitting: gate_up_proj split into separate gate_proj and up_proj
• KV projection splitting: kv_b_proj split into embed_q and unembed_out for MLA

Also Available

• Mistral-Small-4-119B-2603-MLX-4bit — 4-bit version (~62 GB)

Original Model

• mistralai/Mistral-Small-4-119B-2603