supra-50m-instruct-optiq-5bpw-mlx

supra-50m-instruct-optiq-5bpw-mlx

MLX quantization of SupraLabs/Supra-50M-Instruct for Apple Silicon.

Variant: OptiQ mixed-precision (target 5.0 bpw)
Disk size: 42 MB
Quantized by: sahilchachra

About this quantization

Unlike uniform 4-bit quantization (which forces every layer onto the same bit grid and often collapses reasoning at low bit widths), this model was quantized with mlx-optiq using per-layer KL-sensitivity analysis:

1. A small calibration set (32 samples spanning prose, multi-step reasoning, code, and constraint-following instructions) is run through the FP16 reference and through trial quantizations of each layer.
2. The output drift per layer is measured. Layers whose outputs are most affected by quantization (typically the final attention projections, the lm_head, and a few middle blocks) get more bits; layers that tolerate aggressive quantization get fewer.
3. The final assignment hits the target average bits-per-weight while keeping the bits where they matter. This trades off precision unequally so the average comes out near the target (5.0 bits/weight), but the bits that matter most for output fidelity stay high.

Quantization config

• Method: optiq_mixed_precision (mlx-optiq)
• Target bits/weight: 5.0
• Achieved bits/weight: 5.000
• Candidate bits: [4, 6, 8]
• Group size: 64
• Sensitivity reference: bf16
• Calibration: 32-sample 4-domain mix (prose + reasoning + code + constraints)

Per-layer bit allocation

84 quantizable components total. OptiQ allocated bits non-uniformly based on KL sensitivity:

Components kept at 8-bit (most sensitive to quantization):

• model.layers.11.mlp.up_proj
• model.layers.11.mlp.down_proj
• model.layers.11.mlp.gate_proj
• model.layers.11.self_attn.o_proj
• model.layers.11.self_attn.v_proj
• model.layers.11.self_attn.k_proj
• model.layers.11.self_attn.q_proj
• model.layers.0.mlp.up_proj
• model.layers.0.mlp.down_proj
• model.layers.0.mlp.gate_proj
• model.layers.0.self_attn.o_proj
• model.layers.0.self_attn.v_proj
• model.layers.0.self_attn.k_proj
• model.layers.0.self_attn.q_proj

Notice the pattern: the MoE feed-forward gates dominate the top tier. Gate weights pick which experts run for every token — small numerical drift here changes routing decisions and compounds through the rest of the network, so OptiQ pays the bits to keep them precise while compressing the experts themselves more aggressively.

Benchmark results

Evaluated on Apple M4 Pro with MLX. Model loaded once; performance and quality measured in a single pass.

Performance

Quality

Context scaling (decode tok/s)

Usage

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/supra-50m-instruct-optiq-5bpw-mlx")
response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=256, verbose=True)

All variants in this collection

Notes

• Requires Apple Silicon (M1 or later) with MLX
• Benchmarks run on Apple M4 Pro, 24 GB unified memory
• License: see SupraLabs/Supra-50M-Instruct for the original model's license

Original model

See SupraLabs/Supra-50M-Instruct for full model details and intended use.