Inkling-mlx-2bit

Inkling-mlx-2bit (2-bit, omni / multimodal, self-contained MLX)

An MLX 2-bit build of Thinking Machines' Inkling (975B-total / 41B-active MoE), quantized from the BF16 checkpoint. Omni: keeps the text decoder plus the vision (HMLP) and audio (dMel) towers. Self-contained.

We put this repo bundles the loader package (inkling_mlx/), so it runs with just mlx + mlx-lm + transformers.

Community build for large-model-on-Apple-Silicon experiments. ~315 GB in 2-bit -> doesn't fit a single 192 GB Mac resident, so run it via multi-Mac (distributed MLX) or SSD expert-offload (see below — one 192 GB Mac, or even a 64 GB Mac slowly).

Quantization (recipe: experts_only)

• 2-bit (affine int4-style group quant, group size 64) on the routed experts and the vision/audio matmuls (the bulk of the weights).
• Kept bf16 (protected): attention, token/output embeddings, RMSNorms, the router gate, the per-layer short-convolutions, and the relative-position bias.

Usage (self-contained)

# pip install mlx mlx-lm transformers ; download this repo (it includes inkling_mlx/)
from inkling_mlx.load import load
from inkling_mlx.generate import greedy_generate
from transformers import AutoTokenizer

model, config = load("./Inkling-mlx-2bit")
tok = AutoTokenizer.from_pretrained("./Inkling-mlx-2bit", trust_remote_code=True)
ids = tok("The capital of France is")["input_ids"]
print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))

Run on a single Mac with SSD expert-offload

• See image of SSD expert-offload in: github.com/huckiyang/mlx-moe-offload

~315 GB doesn't fit a 192 GB Mac resident but an MoE only fires 6 of 256 experts per token, so you can keep the always-needed weights in RAM (attention, shared experts, embeddings, norms, router, vision/audio towers ≈ 35 GB) and page the routed experts from SSD on demand, letting the OS page cache hold the hot ones. This makes the 2-bit omni build runnable on one 192 GB Mac Studio (and even generates on a 64 GB Mac, slowly).

Tool: github.com/huckiyang/mlx-moe-offload (MIT) — a drop-in over mlx-lm's SwitchGLU. It also answers the mlx-lm feature request ml-explore/mlx-lm#1438.

pip install mlx mlx-lm scipy pillow
git clone https://github.com/huckiyang/mlx-moe-offload && cd mlx-moe-offload && pip install -e .

# 1) one-time: repack the stacked experts into a per-expert SSD store
#    (copies the bundled inkling_mlx/ loader in, so the offload dir is self-contained)
python -m mlx_moe_offload.repack --build /path/Inkling-mlx-2bit --out /path/Inkling-mlx-2bit-offload
OFF=/path/Inkling-mlx-2bit-offload

# 2) generate — text, image, or audio (omni)
python examples/inkling_omni.py --offload-dir $OFF --prompt "The capital of France is"
python examples/inkling_omni.py --offload-dir $OFF --image cat.png --prompt "What is in this image?"
python examples/inkling_omni.py --offload-dir $OFF --audio q.wav   --prompt "Transcribe in English:"

The prompt is prefilled in small --prefill-chunk blocks so only a bounded set of experts is live at once (a lazy full-prompt prefill would page ~all experts per layer and OOM). Decode speed is gated by the routing hit-rate + SSD bandwidth — check store.stats(). At 2-bit, text is solid ("…capital of Japan is Tokyo") while image/audio are recognizable but degraded by quantization; a higher-bit build sharpens multimodal quality.

Sample decoding (on-device, 2-bit + SSD expert-offload, M4 Max 64 GB)

Run entirely on a single 64 GB Mac with the routed experts paged from SSD:

The audio example transcribes clean English speech verbatim — the dMel audio tower + text backbone recover the sentence exactly, even at 2-bit. (Harder / accented clips can drift toward phonetically-similar wordings at 2-bit; a higher-bit build tightens this.)

Throughput is SSD-bound: on this 64 GB Mac (cold cache, expert LRU off) decode runs roughly 0.03–0.4 tok/s, with most of the wall-clock spent in a one-time prefill as the selected experts stream from disk. A 192 GB Mac Studio keeps about half the experts resident in the page cache and should reach the ~6–15 tok/s range.

Attribution

• Weights converted from thinkingmachines/Inkling.

• We made a revised bundled inkling_mlx/ loader is vendored from PipeNetwork/inkling-mlx (Apache-2.0, Copyright 2026 PipeNetwork) and David.

LICENSE + THIRD_PARTY_NOTICES.md are included.