GigaChat3.1-10B-A1.8B-MLX-4bit

GigaChat3.1-10B-A1.8B — MLX 4-bit

First MLX conversion of Sber's GigaChat 3.1. DeepSeek V3 MoE architecture running natively on Apple Silicon.

Specs

Usage

pip install mlx-lm

# Quick generate
mlx_lm.generate --model RockTalk/GigaChat3.1-10B-A1.8B-MLX-4bit --prompt "Explain gradient descent:"

# Chat
mlx_lm.chat --model RockTalk/GigaChat3.1-10B-A1.8B-MLX-4bit

from mlx_lm import load, generate

model, tokenizer = load("RockTalk/GigaChat3.1-10B-A1.8B-MLX-4bit")

messages = [{"role": "user", "content": "What is LoRA fine-tuning?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
response = generate(model, tokenizer, prompt=prompt, max_tokens=500)
print(response)

Conversion Notes

• Converted from ai-sage/GigaChat3.1-10B-A1.8B-bf16
• Multi-token prediction (MTP) head stripped for mlx_lm compatibility (num_nextn_predict_layers set to 0, layer 26 weights removed)
• Tokenizer regex warning is cosmetic and does not affect generation quality
• Quantized with mlx_lm.convert --quantize --q-bits 4 --q-group-size 64

Benchmarks

Tested on M3 Ultra (512GB):

32/32 validation tests passed before upload.

About GigaChat

GigaChat is developed by Sber (Russia's largest bank) through their AI lab ai-sage. It uses the DeepSeek V3 MoE architecture — 64 routed experts with 4 active per token, plus 1 shared expert. The 10B variant is their efficient model, designed for fast inference with minimal memory.

Converted by RockTalk.