Inkling-Small-REAP25-3bE

Inkling-Small-REAP25-3bE

A tiered re-quantization of pipenetwork/Inkling-Small-MLX-REAP25-4bit, itself a REAP-pruned MLX build of thinkingmachines/Inkling-Small. 3-bit routed experts, everything else left at 4-bit. 82GB.

Credit where it belongs: REAP pruning is Cerebras, arXiv:2510.13999; the pruned 4-bit MLX base and the inkling_mlx loader are PipeNetwork; the model is Thinking Machines Lab's. This repo contributes only the expert-tier requant and the measurements below.

Why this exists

Inkling-Small is an unusually capable open multimodal MoE, and the 4-bit REAP build is ~112GB — which fits a 128GB Apple Silicon machine only by leaving no room for a long-context KV cache. The question was simple: do the routed experts survive deeper quantization while attention, embeddings and the shared "sink" experts stay at 4-bit? If yes, you buy back tens of gigabytes of working memory for context, which is the scarce resource on a single machine.

The answer turned out to be interesting in both directions.

Measured (our eval suite, temperature 0 unless noted)

3bE scored 0.97 on intelligence — the best result we have ever recorded on this suite, from any model. 2bE returned a perfect 20/20 on evalplus, which matters because it proves code ability survives 2-bit experts; 3bE's lower code number is an output-duplication artifact, not a weights limitation.

The honest counterpoint: at 2-bit experts, reasoning sometimes fails to terminate (4 parse failures on the intelligence suite). The capability is there; the stopping behaviour degrades.

Status: not production-ready, and the reasons are not the weights

Long-context evaluation is blocked upstream, not by this quantization:

• omlx's prefill preflight uses a static ~250-280KB/token workspace constant versus <1GB of actual KV at 99K, so large prefills are rejected on headroom the model does not need (jundot/omlx#2454)
• the community inkling_mlx checkpoint layout needs engine support (#2451)
• nested-object tool arguments are dropped by the tool-call parser (#2453)

All three are integration-layer and all are open. Until they land, this is a research artifact: excellent on everything that reaches the weights, unproven past ~8K context.

Use

Requires the inkling_mlx loader and an engine with community-layout support. Recommend stop=["<|end_message|>"] to suppress the duplication artifact.