qwen3.5-122b-tq3

qwen3.5-122b-tq3

TurboQuant 3-bit quantized version of Qwen/Qwen3.5-122B-A10B using TurboQuant-MLX.

Documentation: manjunathshiva.github.io/turboquant-mlx: the Qwen3.5-122B-A10B page, getting started, choosing a model for your Mac's memory, guides for serving an OpenAI-compatible API, local coding agents, models bigger than RAM and memory and macOS settings.

Model Details

• Base Model: Qwen/Qwen3.5-122B-A10B (MoE: 122B total params, 10B active)
• Quantization: TurboQuant 3-bit (Hadamard rotation + Lloyd-Max codebook)
• Parameters: ~122B total, ~10B active
• Size: ~50 GB (vs ~240 GB BF16)
• Group Size: 64

<!-- turboquant-plan:begin -->

Will it fit my Mac?

Don't guess, and don't download 54.1 GB to find out — ask first (needs turboquant-mlx-full >= 0.15.1):

turboquant-plan --model manjunathshiva/qwen3.5-122b-tq3

It reads only this repo's safetensors headers over the network (a couple of hundred KB, a few seconds — the weights are never fetched), then projects the peak against your machine and prints the flags to use:

Model
  weights (exact)      54.1 GB
    experts            50.2 GB  (streamable)
    resident           3.9 GB
  KV cache             24.0 KB/token  (hybrid: 12/48 full-attention)

Verdict: <resident | needs a wired-limit raise | streaming>
Recommended: <the exact flags for your Mac>

Planning for a machine you're not sitting at — deciding whether a 16 GB mini or a 64 GB Mac can take this — is what --wired-gb / --ram-gb are for:

turboquant-plan --model manjunathshiva/qwen3.5-122b-tq3 \
    --wired-gb 10.5 --ram-gb 16 --context 8192

turboquant-doctor runs the same projection plus a readiness check; both take --json. The projection is calibrated against real measurements on a 16 GB Mac mini rather than estimated from theory — on the 9.4 GB ternary 35B it predicts a 10.44 GB peak where that machine measures 10.42.

<!-- turboquant-plan:end -->

Requirements

# macOS with Apple Silicon (M1/M2/M3/M4/M5)
pip install "turboquant-mlx-full>=0.4.1"

Quick Start

Generate text

python -m turboquant_mlx.generate \
    --model ~/path/to/qwen3.5-122b-tq3 \
    --prompt "Why is the sky blue? Explain in simple terms." \
    --max-tokens 200 \
    --temp 0.7

With KV Cache Compression

For longer prompts/context, use KV cache compression:

python -m turboquant_mlx.demo_kv \
    --model ~/path/to/qwen3.5-122b-tq3 \
    --prompt "Your long prompt here..." \
    --max-tokens 200 \
    --tq-bits 3

Running on 16 GB RAM (expert streaming)

This 122B model — ~54 GB on disk — runs on a 16 GB Mac mini via expert streaming: for each token only the router-selected experts are paged from disk (LRU-cached), so the 256-expert tensors are never all in memory at once. Resident memory stays well under the machine's GPU wired-memory cap, and output is bit-identical to the fully-resident model. Requires TurboQuant-MLX 0.4.1+.

pip install "turboquant-mlx-full>=0.4.1"

python -m turboquant_mlx.stream.stream_generate \
    --model manjunathshiva/qwen3.5-122b-tq3 \
    --prompt "Explain why the sky is blue." \
    --max-tokens 128 --cache-budget-gb 4

Measured on a base Apple M4 Mac mini, 16 GB:

On a 16 GB machine the binding limit is the Metal GPU wired-memory cap (~10.5 GB), not total RAM — and the expert cache counts against it, so mlx_peak ≈ 5 GB + cache_budget. --cache-budget-gb 4 is the sweet spot (~9 GB peak, safe margin); higher budgets risk a Metal out-of-memory error. Throughput is disk-bandwidth-bound (~10B active params/token), so expect ~1 tok/s on a single mini SSD — slow, but a 122B model genuinely runs on a 16 GB Mac.

Results

How It Works

TurboQuant applies:

1. Hadamard rotation - random +/- 1 scaling to decorrelate weights
2. Lloyd-Max codebook - optimal scalar quantization via k-means
3. Group-wise scaling - per-group float16 scales for precision

This achieves much better quality than standard affine quantization at the same bit-width. The 3-bit on 3-bit (weights + KV cache) works cleanly on 100B+ models which have enough redundancy to absorb stacked noise.

License

Apache 2.0 (same as base model)

Citation

@article{zandieh2025turboquant,
  title={TurboQuant: A Unified Framework for Extremely Low-Bit Weight and KV Cache Quantization},
  author={Zandieh, Amir and Han, Minsik and Dalca, Andre and Shin, Jungwoo and Wang, Brian and Zhang, Yichao and Bordegoni, Matteo and Tian, Yuan and others},
  year={2025}
}