sarvam-105b-mlx-q6

Sarvam-105B MLX Q6

This is an MLX 6-bit affine quantization of sarvamai/sarvam-105b, converted from the public BF16 checkpoint abhinand/sarvam-105b-bf16.

The model is a base/pretrain text-generation model using the sarvam_mla architecture: DeepSeek-V3-style MLA attention plus sigmoid MoE routing. It is not an instruction-tuned chat model.

Why Q6

We tested both q4 and q6 locally on Apple Silicon. The q4 conversion loads and runs, but it is not usable: it collapses to whitespace with probability 1.0 across sanity prompts. The q6 conversion is coherent and passed real-weight validation, so q6 is the published practical checkpoint.

Local validation summary:

Example q6 top-token probes:

Validation

Measured locally on an Apple Silicon M5 Max host with 128 GB unified memory.

• Real BF16 source checkpoint loaded through the MLX sarvam_mla port during conversion.
• q6 generation smoke passed.
• Warm short-context decode measured about 46 tok/s, with about 86 GB peak memory.
• Long-context absorbed-MLA checks ran at 2k, 8k, and 16k context.
• Prefix-cache and prompt-lookup decoding checks passed on suitable workloads.

The q4 collapse was investigated against RoPE, config mapping, and the kv_b_proj sanitize path; those matched the reference implementation. The failure is attributed to naive 4-bit quantization of this large DeepSeek-style MLA-MoE model's activation outliers.

Loading

This artifact includes the custom configuration/modeling files referenced by auto_map. Use MLX tooling with remote/local custom code support as needed.

from mlx_lm import load, generate

model, tokenizer = load("pierrelamy/sarvam-105b-mlx-q6")
response = generate(model, tokenizer, prompt="The capital of France is", max_tokens=32)
print(response)

Provenance

• Base model: sarvamai/sarvam-105b
• BF16 conversion source: abhinand/sarvam-105b-bf16
• Format: MLX safetensors
• Quantization: affine 6-bit, group size 64
• Tensor payload reported by model.safetensors.index.json: 86,202,965,504 bytes
• Parameters reported by model.safetensors.index.json: 106,031,767,424

Caveats

• This is a base model, not an instruct model.
• q4 is intentionally not published because it collapsed in local validation.
• The checkpoint is large: expect roughly 80 GB on disk and about 86 GB peak memory in the measured short-context generation setup.