gemma-4-12B-it-Mixed-8bit

gemma-4-12B-it-Mixed-8bit

Mixed-precision 8-bit quantization of Gemma 4 12B IT — Google's unified multimodal model (text + vision + audio + video), optimized for efficient local inference with minimal quality loss.

Quick Stats

Quantization Strategy

This model uses surgical mixed-precision quantization: only the SwiGLU MLP projection matrices (gate_proj, up_proj, down_proj) in the language model are quantized to 8-bit. All other components remain in BF16:

• ✅ Quantized (8-bit): LLM MLP layers (144 tensors across 48 layers, ~60% of model mass)
• ✅ Protected (BF16): All attention (sliding + full), vision tower, audio embedder, embeddings, LM head, MultimodalEmbedder, all normalization layers, layer_scalars

Rationale: MLP layers are "knowledge lookup" / feature transformation layers that are extremely quantization-tolerant. Attention mechanisms carry reasoning and context-window recall. Vision, audio, and multimodal projection layers are kept in BF16 to preserve modality fidelity.

Model Architecture

Language Model (Text)

• 48 dense layers in pattern [sliding_attention × 5, full_attention × 1] × 8
• Hybrid attention: Sliding window attention (40 layers, head_dim=256) + full global attention (8 layers, head_dim=512)
• Heads: 16 query / 8 KV / 1 global KV
• Hidden size: 3840, Intermediate: 15360
• Vocab: 262,144, Context: 262,144 tokens
• RoPE: Default (θ=10,000) for sliding; Proportional (θ=1,000,000, partial_rotary=0.25) for full attention
• No MoE (dense architecture)

Vision Tower

• ViT with patch_size=16, pooling_kernel=3
• 280 soft tokens per image, projected to 3840-dim
• Position embedding size: 1120

Audio Tower

• Audio embedder: 640-dim embedding, 640 samples per token
• Simple linear projection (no convolutional audio tower in this checkpoint)

Multimodal

• Unified processor: Gemma4UnifiedProcessor handling text, images, audio, and video
• Special tokens: <|image|>, <|audio|>, <|video|> with BOI/EOI, BOA/EOA delimiters
• MultimodalEmbedder projects vision and audio into the shared 3840-dim text space

MTP Speculative Decoding

This model supports Multi-Token Prediction (MTP) speculative decoding via the companion assistant/drafter model. This model achieves 20-30% faster token generation with gemma-4-12B-it-assistant in BF16 format.

Usage

oMLX (Recommended)

Loads natively with MTP and vision support.

MLX / mlx-vlm

from mlx_vlm.utils import load

model, processor = load("your-username/gemma-4-12B-it-Mixed-8bit", lazy=True)
# Model loads in ~15 GB vs ~24 GB for BF16

Files

Quantization Details

Reproducing

# Quantize from the BF16 base model
mlx-env/bin/python quantization_scripts/mixed_quantize_gemma4_12b.py \
    --bits 8 --group-size 64

The script:

1. Lazily loads the BF16 checkpoint via mlx_vlm.utils.load()
2. Applies 8-bit quantization to MLP projections only (via class_predicate filter)
3. Saves config with quantization metadata
4. Flattens, evaluates, and shards to 5 GB safetensors

Limitations

• Mixed quantization may show minor quality degradation on edge cases compared to BF16
• The vision tower, audio embedder, and multimodal projector are kept in BF16 — they are not quantized in this version
• Audio support uses a simple linear projection (no convolutional audio tower); complex audio understanding may be limited
• MTP speculative decoding requires the companion assistant model to be loaded alongside
• The base model uses transformers_version: 5.10.0.dev0 — some fields may not be recognized by stable transformers releases yet

License

This model inherits the Apache 2.0 license of the base google/gemma-4-12b-it.