Ling-3.0-Flash-4.6bpw-MLX

Ling-3.0-Flash-4.6bpw-MLX

A mixed-precision MLX quantization of inclusionAI/Ling-3.0-flash, sized for local inference on Apple Silicon. ~67GB, down from the ~237GB BF16 original, at an effective 4.616 bits/weight average.

Quantization scheme

Not a uniform bit-width — the name reflects the real average, not a round number:

Affine (round-to-nearest) quantization, group_size=64, no calibration data required. Built with a custom streaming pipeline (lazy-loads the BF16 teacher and quantizes shard-by-shard, so the full 237GB model is never resident) — tooling published at True2456/streaming-dwq-mlx.

Why 4-bit routed experts specifically

Tested routed-expert bit widths from 2 to 6-bit against the BF16 teacher, using realistic chunked prefill (not a single-mega-chunk test, which this specific model's hybrid KDA/MLA architecture is unusually sensitive to — see the linked repo's findings doc if you're building your own quantization of this model family). Results, teacher-vs-quantized KL divergence sampled at 64 positions across a 16K-token document:

4-bit was the chosen tradeoff: meaningfully smaller than a straightforward 5-bit or 8-bit quant, with no measured degradation.

Real benchmark validation

KL divergence against a teacher is a distributional proxy, not a task score — so this was also run through real accuracy benchmarks (mmlu, gsm8k, humaneval) against a different original quantized-routed 5-bit-average build:

At these sample sizes, a 2-4 question swing is within normal binomial noise — statistically indistinguishable from parity.

Important: two upstream mlx-lm bugs this model needs patched

mlx-lm's bailing_hybrid implementation (the architecture family Ling-3.0-flash belongs to) is missing two pieces of Ling-specific logic that its own config.json specifies. Neither is optional — running without them costs real accuracy, silently, with no error or crash to signal it:

1. SwiGLU clamp (expert_swiglu_limit_list / share_expert_swiglu_limit_list, layers 34-41). Measured cost of running unclamped: 88.41% → 71.34% HumanEval (17 points).
2. KDA safe-gate clamp (kda_safe_gate: true, kda_lower_bound: -5.0). Some mlx-lm builds never read these config fields; the HF reference implementation does. Same category as (1): silent, real accuracy cost.

Both patches: bailing_swiglu_clamp.py, kda_safe_gate_patch.py. If you're loading this model through a runtime that doesn't already apply equivalent fixes, apply these first or expect degraded output.

Usage

Requires an mlx-lm build with bailing_hybrid support and the two patches above applied (see the linked repo for details).

from mlx_lm import load, generate

model, tokenizer = load("True2456/Ling-3.0-Flash-4.6bpw-MLX")
# apply the SwiGLU clamp + KDA safe-gate patches before/at load time -- see
# https://github.com/True2456/streaming-dwq-mlx for the patch modules

response = generate(model, tokenizer, prompt="Hello!", max_tokens=256)
print(response)

License

MIT, inherited from the base model.