Downloads · 30 days
204
6% of all-time downloads
TheDrainFlorist/Qwen3.6-35B-A3B-VQ-5.4bpw
Qwen3.6-35B-A3B-VQ-5.4bpw is a text generation model from TheDrainFlorist. Use it when you need the model to write or continue text. It is set up for mlx. The card lists the license as apache-2.0.
22.2 GiB — the closest to bf16 in this collection.
Downloads · 30 days
204
6% of all-time downloads
All-time downloads
3.4K
Public
Parameters
8.4B
24.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors24.6 GB · 100%
How the weights are stored.
U327.5B · 88%
From the Hugging Face model README
22.2 GiB — the closest to bf16 in this collection.
A vector-quantized build of
Qwen3.6-35B-A3B for Apple
Silicon. Stock mlx-lm, no patches — the VQ runtime ships inside the
checkpoint as model.py.
This repo now ships mtp-head-q6.safetensors, a 0.70 GiB multi-token-prediction
draft head for speculative decoding. The weights and the bundled model.py
are unchanged. The head is Qwen3.6-35B-A3B's own mtp.* layer from the bf16
release (stripped from the common MLX conversions), quantized to 6-bit, group 32.
Knurlogic drafts with it automatically; see Speculative decoding.
The exo --mtp note in the 2026-09 entry below is superseded: exo is no longer the serving path, and no head shipped until this revision.
The bundled model.py is updated to the current VQ runtime; the weights are unchanged. The run, multi-Mac, drafting and vision instructions now use Knurlogic.
KL to this family's own bf16 teacher (cached top-64), 12288 tokens, on all three house corpora — prose, public code and literary — every rung scored on the same cache and the same positions, through a streamed pass validated against a direct full-model forward to all printed decimals.
| build | GiB | prose | code | literary | mean | top-1 |
|---|---|---|---|---|---|---|
| VQ-3.4bpw | 13.8 | 73.3 | 520.2 | 207.5 | 267.0 | 89.4% |
| VQ-3.8bpw | 15.7 | 44.5 | 423.6 | 101.4 | 189.8 | 91.4% |
| VQ-4.6bpw | 18.7 | 33.7 | 409.2 | 58.3 | 167.1 | 92.7% |
| VQ-5.4bpw | 22.2 | 21.6 | 345.1 | 33.5 | 133.4 | 94.0% |
Millinats per token; lower is closer to bf16. These rows are VQ rungs only and are not comparable to the affine-comparator table further down, which was measured prose-only at 2048 tokens on an earlier instrument.
Rank by KL, not perplexity. On this family perplexity is not monotone in quality: rungs that are measurably closer to the teacher can read a higher perplexity, because a damaged model can score better than bf16 on a finite sample. KL on cached teacher logits does not have that failure mode.
Runtime refresh. The bundled model.py is updated so downloaders run
exactly the code that was benchmarked; dense bundles now carry both runtimes.
mtp-head-q6.safetensors, the sidecar works with the exo fork branch
mtp-stage1 (github.com/noahzelezny/exo) — launch each node with
exo --mtp (or set EXO_MTP=1). Note: with MTP enabled, exo serves
requests sequentially (the batch engine has no MTP path), so leave it
off for concurrent / multi-agent workloads.Bundle refresh.
The bundled model.py runtime is updated as part of
the 2026-09 refresh shared across the VQ lineup:
mlx-lm and mlx_vlm.model.py is replaced, and the prior revision stays pinnable by
commit hash.See also the 2026-09-17 entry in the changelog above, which places this rung against the rest of its family on the three-corpus KL instrument. The table below is the earlier prose-only measurement and is not comparable to it row-for-row.
Referee corpus, 2048-token windows, scored against the bf16 teacher on the
same files with an unmodified mlx-lm. All sizes include the bf16 vision
tower, which every build here carries:
| build | size | KL to bf16 (mnats/tok) | top-1 agreement |
|---|---|---|---|
| bf16 | 65.4 GiB | 0 | 100% |
| mlx-community 8-bit | 35.1 GiB | 7.4 | 96.18% |
| this model | 22.2 GiB | 28.1 | 92.22% |
| mlx-community 4-bit | 19.0 GiB | 78.6 | 85.61% |
This build sits between the two community rungs and closer to the 8-bit than to the 4-bit — 28.1 millinats against their 78.6 and 7.4. Interpolating the affine frontier between the 4-bit and a 6-bit build to this exact size gives 38.7 millinats; this build measures 28.1, a factor of 1.4 below the line.
Two independent fits of this recipe were scored, at 28.141 and 27.927 millinats — the only build in this collection measured twice. The spread between them, 0.214 millinats, is this family's fit-to-fit noise floor, and the margin above is roughly fifty times it. The number quoted in the table is one artifact, the one published here, not an average of the two.
Perplexity was not measured on this artifact. The other releases in this family carry a perplexity column; this build was scored on KL and top-1 agreement, which is the instrument this family is ranked by. Reporting a sibling's perplexity here would be a substitution this project does not make.
One Mac Studio M3 Ultra, vqlab speed-pair: a fresh process per run,
alternating with the reference, 3 pairs, 2048-token prompt, 128 generated
tokens, no drafting.
| this model | Qwen3.6-35B-A3B-8bit | |
|---|---|---|
| decode | 48.8 tok/s (0.84×) | 58.0 tok/s |
| prefill | 2151 tok/s (0.81×) | 2852 tok/s |
| peak memory | 24.6 GB | 38.8 GB |
The reference's own decode speed swung between pairs in this run, so treat the decode ratio as approximate.
Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.
Comfortable on a 32 GB machine.
Requires a Mac with Apple Silicon (macOS, MLX).
With Knurlogic, which works out the settings and checks the model fits before loading it:
pip install knurlogic
hf download TheDrainFlorist/Qwen3.6-35B-A3B-VQ-5.4bpw --local-dir ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-5.4bpw
knurlogic serve ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-5.4bpw
Chat at http://127.0.0.1:8080/, or point any OpenAI-, Anthropic- or
Ollama-compatible client at it (model local). knurlogic ui opens a page
with a Launch button for every model on disk instead.
Images work through Knurlogic directly; mlx-vlm is not needed.
The VQ runtime ships inside this repo as model.py (declared by
model_file in config.json), and Knurlogic runs the file the model ships.
Knurlogic drafts with mtp-head-q6.safetensors by default when it sits beside
the weights (a full download includes it), and switches between drafting and
plain decoding by measured cost.
knurlogic serve <model> --set KNURLOGIC_MTP=off, or MTP Off in
the load panel.--set KNURLOGIC_MTP_DYNAMIC=off.pip install -U knurlogic).Measured on this build with this head (M4 Max, one Mac, greedy, thinking
off, 400 tokens, 3 prompts; Knurlogic main + DSpark work, mlx 0.32.3):
| plain | drafting | |
|---|---|---|
| decode | 57.9–60.4 tok/s | 87.1–95.4 tok/s (1.53×) |
| acceptance | 0.872 (524 of 601 drafts) |
Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.
Without Knurlogic:
pip install 'mlx-lm>=0.31.3'
python -m mlx_lm generate \
--model TheDrainFlorist/Qwen3.6-35B-A3B-VQ-5.4bpw \
--prompt "Write a Python function that ..." \
--max-tokens 512
Uniform vector quantization of the experts at d=2, K=1024 — each 2-weight subvector stores one 10-bit index into a per-tensor 1024-entry fp16 codebook, with an fp16 scale per (row, 64 weights). Non-expert tensors are 8-bit.
Halving the subvector dimension is what buys the quality here. At d=4 every additional quarter-bit costs a doubling of the codebook, so the d4 line saturates; spending the same bits at d=2 keeps paying. There is no depth schedule and no mixed allocation — every expert tensor uses the same geometry.
Codebooks are fit in pure weight space — k-means over the weight subvectors, no Hessian, no activation statistics, no calibration corpus.
The fit is not seeded. k-means draws an unseeded subsample, so this artifact is reproducible in recipe and geometry but not bit-for-bit. That is why two fits were scored rather than one.
Every tensor was decoded from the published artifact and compared against the bf16 source; no tensor exceeds 3x the artifact's own median reconstruction error (median 0.0411). The bundled runtime was exercised as the executing copy in a stock venv, not merely present in the folder.
Vision tower grafted in (333 tensors, 0.83 GiB); the text path is unchanged, verified against the same KL cache.
The method, the full model ladder, the negative results, and the measurement rules behind every number here: Data-Free Vector Quantization Beats Affine Quantization at Matched Bytes Below 6 Bits (CC BY 4.0) · code: VQLab · web version: Space
VQLab and these artifacts are built and released independently — the fits, the measurement harness and the published rungs are one person's compute and time. If they are useful to you:
Base model: Qwen/Qwen3.6-35B-A3B — Apache-2.0. This is a quantized derivative and inherits that license; using it means accepting the base model's terms. Quantization: TheDrainFlorist, 2026.
Built with MLX and VQLab.