lfm2-5-8b-a1b-optiq-5bpw-mlx

lfm2-5-8b-a1b-optiq-5bpw-mlx

MLX quantization of LiquidAI/LFM2.5-8B-A1B for Apple Silicon.

Variant: OptiQ mixed-precision (target 5.0 bpw)
Disk size: 8469 MB
Quantized by: sahilchachra

About this quantization

Unlike uniform 4-bit quantization (which forces every layer onto the same bit grid and often collapses reasoning at low bit widths), this model was quantized with mlx-optiq using per-layer KL-sensitivity analysis:

1. A small calibration set (32 samples spanning prose, multi-step reasoning, code, and constraint-following instructions) is run through the FP16 reference and through trial quantizations of each layer.
2. The output drift per layer is measured. Layers whose outputs are most affected by quantization (typically the final attention projections, the lm_head, and a few middle blocks) get more bits; layers that tolerate aggressive quantization get fewer.
3. The final assignment hits the target average bits-per-weight while keeping the bits where they matter. This trades off precision unequally so the average comes out near the target (5.0 bits/weight), but the bits that matter most for output fidelity stay high.

Quantization config

• Method: optiq_mixed_precision (mlx-optiq)
• Target bits/weight: 5.0
• Achieved bits/weight: 5.040
• Candidate bits: [3, 4, 6, 8]
• Group size: 64
• Sensitivity reference: uniform_4bit
• Calibration: 32-sample 4-domain mix (prose + reasoning + code + constraints)

Note on architecture: this base model mixes attention/MLP blocks with convolutional / state-space blocks. OptiQ's per-layer sweep only ranks the attention + feed-forward weights — the conv blocks stay at their original precision. The achieved bits/weight above is the average over the quantized components; the on-disk model size is correspondingly larger than a fully-quantized model at the same target bpw would be.

Per-layer bit allocation

82 quantizable components total. OptiQ allocated bits non-uniformly based on KL sensitivity:

Components kept at 8-bit (most sensitive to quantization):

• model.layers.23.feed_forward.gate
• model.layers.22.feed_forward.gate
• model.layers.21.feed_forward.gate
• model.layers.20.feed_forward.gate
• model.layers.19.feed_forward.gate
• model.layers.18.feed_forward.gate
• model.layers.17.feed_forward.gate
• model.layers.16.feed_forward.gate
• model.layers.15.feed_forward.gate
• model.layers.14.feed_forward.gate
• model.layers.13.feed_forward.gate
• model.layers.12.feed_forward.gate
• model.layers.11.feed_forward.gate
• model.layers.10.feed_forward.gate
• model.layers.9.feed_forward.gate
• model.layers.8.feed_forward.gate
• model.layers.7.feed_forward.gate
• model.layers.6.feed_forward.gate
• model.layers.5.feed_forward.gate
• model.layers.4.feed_forward.gate
• model.layers.3.feed_forward.gate
• model.layers.2.feed_forward.gate
• model.layers.0.conv.out_proj
• model.layers.0.conv.in_proj

Notice the pattern: the MoE feed-forward gates dominate the top tier. Gate weights pick which experts run for every token — small numerical drift here changes routing decisions and compounds through the rest of the network, so OptiQ pays the bits to keep them precise while compressing the experts themselves more aggressively.

Benchmark results

Evaluated on Apple M5 Pro with MLX. Model loaded once; performance and quality measured in a single pass.

Performance

Warmed, short-prompt, chat-templated, thinking disabled. Represents steady-state decode for typical chat use; long thinking traces will be slower due to KV-cache growth.

Quality

MATH-500 per-level accuracy

Context scaling (decode tok/s)

Usage

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/lfm2-5-8b-a1b-optiq-5bpw-mlx")
response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=256, verbose=True)

All variants in this collection

Notes

• Requires Apple Silicon (M1 or later) with MLX
• Benchmarks run on Apple M5 Pro, 24 GB unified memory
• License: see LiquidAI/LFM2.5-8B-A1B for the original model's license

Original model

See LiquidAI/LFM2.5-8B-A1B for full model details and intended use.