Downloads · 30 days
214
6% of all-time downloads
TheDrainFlorist/Qwen3.6-35B-A3B-VQ-3.8bpw
Qwen3.6-35B-A3B-VQ-3.8bpw is a text generation model from TheDrainFlorist. Use it when you need the model to write or continue text. It is set up for mlx. The card lists the license as apache-2.0.
15.7 GiB — smaller than the community 4-bit and a third closer to bf16.
Downloads · 30 days
214
6% of all-time downloads
All-time downloads
3.8K
Public
Parameters
6.7B
17.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors17.6 GB · 100%
How the weights are stored.
U325.7B · 85%
From the Hugging Face model README
15.7 GiB — smaller than the community 4-bit and a third closer to bf16.
A vector-quantized build of
Qwen3.6-35B-A3B for Apple
Silicon. Stock mlx-lm, no patches — the VQ runtime ships inside the
checkpoint as model.py.
This repo now ships mtp-head-q6.safetensors, a 0.70 GiB multi-token-prediction
draft head for speculative decoding. The weights and the bundled model.py
are unchanged. The head is Qwen3.6-35B-A3B's own mtp.* layer from the bf16
release (stripped from the common MLX conversions), quantized to 6-bit, group 32.
Knurlogic drafts with it automatically; see Speculative decoding.
The exo --mtp note in the 2026-09 entry below is superseded: exo is no longer the serving path, and no head shipped until this revision.
The bundled model.py is updated to the current VQ runtime; the weights are unchanged. The run, multi-Mac, drafting and vision instructions now use Knurlogic.
KL to this family's own bf16 teacher (cached top-64), 12288 tokens, on all three house corpora — prose, public code and literary — every rung scored on the same cache and the same positions, through a streamed pass validated against a direct full-model forward to all printed decimals.
| build | GiB | prose | code | literary | mean | top-1 |
|---|---|---|---|---|---|---|
| VQ-3.4bpw | 13.8 | 73.3 | 520.2 | 207.5 | 267.0 | 89.4% |
| VQ-3.8bpw | 15.7 | 44.5 | 423.6 | 101.4 | 189.8 | 91.4% |
| VQ-4.6bpw | 18.7 | 33.7 | 409.2 | 58.3 | 167.1 | 92.7% |
| VQ-5.4bpw | 22.2 | 21.6 | 345.1 | 33.5 | 133.4 | 94.0% |
Millinats per token; lower is closer to bf16. These rows are VQ rungs only and are not comparable to the affine-comparator table further down, which was measured prose-only at 2048 tokens on an earlier instrument.
Rank by KL, not perplexity. On this family perplexity is not monotone in quality: rungs that are measurably closer to the teacher can read a higher perplexity, because a damaged model can score better than bf16 on a finite sample. KL on cached teacher logits does not have that failure mode.
Runtime refresh. The bundled model.py is updated so downloaders run
exactly the code that was benchmarked; dense bundles now carry both runtimes.
mtp-head-q6.safetensors, the sidecar works with the exo fork branch
mtp-stage1 (github.com/noahzelezny/exo) — launch each node with
exo --mtp (or set EXO_MTP=1). Note: with MTP enabled, exo serves
requests sequentially (the batch engine has no MTP path), so leave it
off for concurrent / multi-agent workloads.Bundle refresh.
The bundled model.py runtime is updated as part of
the 2026-09 refresh shared across the VQ lineup:
mlx-lm and mlx_vlm.model.py is replaced, and the prior revision stays pinnable by
commit hash.See also the 2026-09-17 entry in the changelog above, which places this rung against the rest of its family on the three-corpus KL instrument. The table below is the earlier prose-only measurement and is not comparable to it row-for-row.
Referee corpus, 2048-token windows, scored against the bf16 teacher on the
same files with an unmodified mlx-lm. All sizes include the bf16 vision
tower, which every build here carries:
| build | size | KL to bf16 (mnats/tok) | top-1 agreement | perplexity |
|---|---|---|---|---|
| bf16 | 65.4 GiB | 0 | 100% | 4.7215 |
| mlx-community 8-bit | 35.1 GiB | 7.4 | 96.18% | 4.7150 |
| this model | 15.7 GiB | 53.0 | 89.55% | 4.7090 |
| mlx-community 4-bit | 19.0 GiB | 78.6 | 85.61% | 4.9154 |
33% less divergence than the 4-bit, at 3.3 GiB smaller. That is the comparison this build exists for.
Read the perplexity column carefully. At 4.7090 this build sits slightly below the bf16 teacher, and the 4-bit sits above it. That does not make it better than bf16 — perplexity is an aggregate over finite text and absorbs offsetting errors in both directions. KL measures distance to the teacher's distribution directly and does not. Treat anything within about 1% of the teacher as indistinguishable, and rank by KL.
Note the two metrics disagree in emphasis: KL and perplexity both favor this build over the 4-bit, while top-1 agreement is 89.55% against the 8-bit's 96.18%. For a 256-expert MoE that is expected — discrete routing flips swap one plausible token for another, which perplexity absorbs and argmax agreement punishes. Both numbers are honest; quote both.
One Mac Studio M3 Ultra, vqlab speed-pair: a fresh process per run,
alternating with the reference, 3 pairs, 2048-token prompt, 128 generated
tokens, no drafting.
| this model | Qwen3.6-35B-A3B-8bit | |
|---|---|---|
| decode | 62.3 tok/s (1.06×) | 58.0 tok/s |
| prefill | 2401 tok/s (0.87×) | 2743 tok/s |
| peak memory | 17.5 GB | 38.8 GB |
Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.
Comfortable on a 24 GB machine; workable on 16 GB at short context.
Requires a Mac with Apple Silicon (macOS, MLX).
With Knurlogic, which works out the settings and checks the model fits before loading it:
pip install knurlogic
hf download TheDrainFlorist/Qwen3.6-35B-A3B-VQ-3.8bpw --local-dir ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-3.8bpw
knurlogic serve ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-3.8bpw
Chat at http://127.0.0.1:8080/, or point any OpenAI-, Anthropic- or
Ollama-compatible client at it (model local). knurlogic ui opens a page
with a Launch button for every model on disk instead.
Images work through Knurlogic directly; mlx-vlm is not needed.
The VQ runtime ships inside this repo as model.py (declared by
model_file in config.json), and Knurlogic runs the file the model ships.
Knurlogic drafts with mtp-head-q6.safetensors by default when it sits beside
the weights (a full download includes it), and switches between drafting and
plain decoding by measured cost.
knurlogic serve <model> --set KNURLOGIC_MTP=off, or MTP Off in
the load panel.--set KNURLOGIC_MTP_DYNAMIC=off.pip install -U knurlogic).Measured on this build with this head (M4 Max, one Mac, greedy, thinking
off, 400 tokens, 3 prompts; Knurlogic main + DSpark work, mlx 0.32.3):
| plain | drafting | |
|---|---|---|
| decode | 71.5–72.1 tok/s | 93.1–103.2 tok/s (1.37×) |
| acceptance | 0.879 (523 of 595 drafts) |
Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.
Without Knurlogic:
pip install 'mlx-lm>=0.31.3'
python -m mlx_lm generate \
--model TheDrainFlorist/Qwen3.6-35B-A3B-VQ-3.8bpw \
--prompt "Write a Python function that ..." \
--max-tokens 512
Uniform vector quantization of the experts at d=4, K=8192 — each 4-weight subvector stores one 13-bit index into a per-tensor 8192-entry fp16 codebook, with an fp16 scale per (row, 64 weights). Non-expert tensors are 8-bit. Experts are ≈92% of the model, so that is where the bytes are.
There is no depth schedule and no mixed allocation: every expert tensor uses the same geometry. Flat rungs are the reference points in this lineup because no mixed-allocation build we measured beat the flat rung at its own size.
Codebooks are fit in pure weight space — k-means over the weight subvectors, no Hessian, no activation statistics, no calibration corpus.
The fit is not seeded. k-means draws an unseeded subsample, so this artifact is reproducible in recipe and geometry but not bit-for-bit. Margins are therefore quoted against a measured fit-to-fit floor rather than against a repeated build; on this family that floor is 0.214 mnats, so the 25-mnat gap to the 4-bit is far outside it.
Every tensor was decoded from the published artifact and compared against the bf16 source; no tensor exceeds 3x the artifact's own median reconstruction error. The packed artifact reproduced its unpacked twin's KL score to every printed digit. The bundled runtime was exercised as the executing copy in a stock venv, not merely present in the folder.
Vision tower grafted in (333 tensors, 0.83 GiB); the text path is unchanged, verified against the same KL cache.
The method, the full model ladder, the negative results, and the measurement rules behind every number here: Data-Free Vector Quantization Beats Affine Quantization at Matched Bytes Below 6 Bits (CC BY 4.0) · code: VQLab · web version: Space
VQLab and these artifacts are built and released independently — the fits, the measurement harness and the published rungs are one person's compute and time. If they are useful to you:
Base model: Qwen/Qwen3.6-35B-A3B — Apache-2.0. This is a quantized derivative and inherits that license; using it means accepting the base model's terms. Quantization: TheDrainFlorist, 2026.
Built with MLX and VQLab.