Downloads · 30 days
151
2% of all-time downloads
alexcovo/qwen35-9b-mlx-turboquant-tq3
qwen35-9b-mlx-turboquant-tq3 is a text generation model from alexcovo. Use it when you need the model to write or continue text. It is set up for mlx. The card lists the license as apache-2.0.
This repo packages the current best TurboQuant runtime recipe we have measured for mlx-community/Qwen3.5-9B-MLX-4bit: TurboQuant 3-bit KV-cache compression (TQ3) on Apple Silicon.
Downloads · 30 days
151
2% of all-time downloads
All-time downloads
6.2K
Public
Parameters
9.4B
6 GB on disk
Likes
4
Public
Click a slice to open those files.
.safetensors6 GB · 100%
How the weights are stored.
U329B · 95%
From the Hugging Face model README
This repo packages the current best TurboQuant runtime recipe we have measured for mlx-community/Qwen3.5-9B-MLX-4bit: TurboQuant 3-bit KV-cache compression (TQ3) on Apple Silicon.
Short version:
This repo does:
mlx-community/Qwen3.5-9B-MLX-4bitKVCache slots with TurboQuantKVCache(bits=3)This repo does not:
That distinction matters. TurboQuant changes the inference-time cache representation, not the underlying model parameters.
For the measured Qwen3.5-9B MLX workload, TQ3 is the current best operating point on the speed/compression frontier:
If you want the cleanest current recommendation rather than a menu of half-baked variants, this is it.
Requirements:
git or hfRecommended hardware:
Option 1: clone the full bundle from Hugging Face
git lfs install
git clone https://huggingface.co/alexcovo/qwen35-9b-mlx-turboquant-tq3
cd qwen35-9b-mlx-turboquant-tq3
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install mlx mlx-lm
pip install -e .
python examples/run_qwen35_tq3.py
Option 2: download without git
hf download alexcovo/qwen35-9b-mlx-turboquant-tq3 --local-dir qwen35-9b-mlx-turboquant-tq3
cd qwen35-9b-mlx-turboquant-tq3
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install mlx mlx-lm
pip install -e .
python examples/run_qwen35_tq3.py
The example script loads the model from the local repo directory, so the weights on disk are used directly.
Smoke test:
python -c "import mlx.core as mx; from pathlib import Path; from mlx_lm import load; from mlx_lm.models.cache import KVCache; from turboquant_mlx import TurboQuantKVCache, apply_patch; root=Path('.').resolve(); model, tokenizer = load(str(root)); apply_patch(); cache=[TurboQuantKVCache(bits=3) if isinstance(c, KVCache) else c for c in model.make_cache()]; logits=model(mx.array(tokenizer.encode('TurboQuant smoke test.'))[None], cache=cache); mx.eval(logits); token=mx.argmax(logits[:, -1, :], axis=-1); logits=model(token.reshape(1, 1), cache=cache); mx.eval(logits); print('TQ3 local bundle OK')"
Troubleshooting:
git clone fetches pointer files instead of weights, run git lfs install first and clone again.hf is missing, install it with pip install -U "huggingface_hub[cli]".pip install -e . fails, make sure you are inside the downloaded repo root.from pathlib import Path
import mlx.core as mx
from mlx_lm import load
from mlx_lm.models.cache import KVCache
from turboquant_mlx import TurboQuantKVCache, apply_patch
repo_root = Path('.').resolve()
model, tokenizer = load(str(repo_root))
apply_patch()
cache = []
for c in model.make_cache():
cache.append(TurboQuantKVCache(bits=3) if isinstance(c, KVCache) else c)
prompt = "Explain why KV-cache compression matters for long-context decoding."
input_ids = mx.array(tokenizer.encode(prompt))[None]
logits = model(input_ids, cache=cache)
mx.eval(logits)
Full runnable example:
examples/run_qwen35_tq3.pyTurboQuant compresses the KV cache at inference time using:
The goal is simple: reduce KV-cache footprint while keeping decode throughput and output quality on the useful side of the tradeoff.

Representative medians from the publication-grade sweep:
| Prompt scale | FP16 tok/s | TQ4 tok/s | TQ3 tok/s | TQ4 cache | TQ3 cache |
|---|---|---|---|---|---|
| 1.00x | 52.90 | 51.58 | 51.67 | 56.2 MB | 50.6 MB |
| 1.50x | 43.42 | 48.71 | 48.65 | 71.4 MB | 63.0 MB |
| 2.25x | 47.97 | 45.74 | 48.26 | 94.1 MB | 81.5 MB |
| 3.00x | 46.62 | 45.49 | 45.35 | 116.8 MB | 100.1 MB |
Interpretation:
TQ3 is the current best speed/compression operating point.examples/run_qwen35_tq3.py: minimal runnable examplebenchmarks/README.md: benchmark summarybenchmarks/summary_rows.tsv: median benchmark tablebenchmarks/raw_rows.tsv: raw benchmark rowsbenchmarks/run_metadata.json: run metadataassets/curve.png / assets/curve.svg: chart artifactsImplementation and ongoing development live here:
Benchmark driver used for this sweep:
benchmarks/qwen35_9b_publication_curve.py in the source repoThis repository is intentionally explicit: it packages a practical runtime recipe and the supporting evidence, not a new set of model weights.