Qwable-v2-oQ4e-DWQ-MTP-Vision-MLX

Qwable-v2-oQ4e-DWQ-MTP-Vision-MLX

A 4-bit MLX quantization of lordx64/Qwable-v2 — vision + text, with an embedded MTP head for speculative decoding. Built with a two-stage recipe: imatrix mixed-precision (oQ4e) → Distilled Weight Quantization (DWQ), the strongest 4-bit path we could measure on this model.

⚠️ These are quantized weights. All capability comes from the base model lordx64/Qwable-v2 — please star/cite it first. This repo's contribution is the quantization + MTP/vision packaging and its fidelity/speed validation.

Model lineage

Qwen/Qwen3.6-35B-A3B                          (base, 35B MoE · 256 experts · ~3B active · 262k ctx)
  └─ lordx64/…-Claude-4.7-Opus-Reasoning-Distilled   (SFT distill of Claude Opus 4.7 reasoning)
       └─ lordx64/Qwable-v2                            (+ Claude Fable-5 agentic / tool-use LoRA)
            └─ THIS REPO: oQ4e (imatrix) → DWQ 4-bit MLX + MTP + vision

Qwable-v2 is an open-weights agentic coding model: it thinks in explicit <think>…</think> chains-of-thought (from the Opus-4.7 prior) and acts like a Claude-Code-style agent — emitting <tool_use> XML with real Claude Code tool names (Read, Edit, Bash) and correct field signatures. The agentic XML is system-prompt-conditional (reliable with an agent-style system prompt or after a <tool_result> turn; bare prompts fall back to prose).

• Architecture: Qwen3.6-35B-A3B — Mixture-of-Experts, 256 experts (8 routed + 1 shared), ~3B active params/token, up to 262,144-token context.
• This repo: imatrix 4-bit quant, DWQ-distilled, with MTP and the vision tower preserved.

Quantization: oQ4e (imatrix) → DWQ, 4-bit

A two-stage learned quant, not a plain round-to-nearest 4-bit:

1. oQ4e — imatrix mixed-precision. oMLX's enhanced quantizer builds an importance matrix from ~1k calibration activations (sized for MoE expert coverage) and allocates bits per-tensor by sensitivity. Base 4-bit affine (group size 64), with sensitive tensors promoted: 201 → 8-bit, 105 → 5-bit, 7 → 6-bit (the rest stay 4-bit).
2. DWQ — distillation. The oQ4e backbone's sub-8-bit affine scales/biases are then gradient-distilled toward an 8-bit teacher of the same model (KL on top-1024 logits, temperature 2.0, lr 1e-6, seq 512), recovering fidelity the quant grid loses.

This is a true imatrix → DWQ build — distillation applied on top of a sensitivity-mixed backbone, rather than on a flat uniform-4-bit student. On this model it is the best-fidelity 4-bit we measured (see below).

Evaluation

Quant fidelity — validation KL-to-8-bit-teacher (lower = better)

Measured identically across recipes on the held-out calibration split (same seed / temperature / teacher targets), so the numbers are directly comparable:

imatrix alone (oQ4e) already beats oQ4+DWQ; adding DWQ on top squeezes out the rest.

Capability benchmarks — measured on this quant

Run by the uploader on this 4-bit build (served via oMLX); reported to show that quantization preserves base capability. Confirm harness/shot settings against your own eval before citing as official base-model numbers.

For the base model's own agentic evals (SWE-bench Lite, etc.), see the Qwable-v2 card — several suites are still in progress there.

Speed (M4 Pro, 48 GB, oMLX 0.5.0, MTP on)

Why the vision build is faster: oMLX routes a model with a vision tower to its VLM engine, which runs this qwen3_5_moe architecture natively with Lightning MTP (custom depth-k speculative-decoding kernels). The text-only sibling routes to the LLM engine, which uses a fallback decode path for this VLM-native architecture. The backbone and MTP weights are byte-identical between the two builds — only decode speed differs, not accuracy. For text tasks this build serves the vision tower unused, so it's a strict speed win.

Sibling repos (same base, pick your trade-off)

Both quant builds share the same distilled backbone + MTP head (byte-identical) — identical task quality; they differ only by the vision tower and thus the serving engine (speed).

How to run

These are MLX weights (Apple Silicon). The tested serving path is oMLX ≥ 0.5.0, which supports this model's native MTP speculative decoding and vision out of the box. This repo includes the vision tower (vision_tower.*, bf16) — it accepts images as well as text.

# 1. place the folder in your oMLX models directory
mv Qwable-v2-oQ4e-DWQ-MTP-Vision-MLX ~/.omlx/models/

# 2. enable MTP once (VLM path), then call the OpenAI-compatible API
curl -X PUT http://127.0.0.1:8003/admin/api/models/Qwable-v2-oQ4e-DWQ-MTP-Vision-MLX/settings \
  -H "Authorization: Bearer sk-local" -H "Content-Type: application/json" \
  -d '{"mtp_enabled": true, "vlm_mtp_enabled": true}'

curl -X POST http://127.0.0.1:8003/v1/chat/completions \
  -H "Authorization: Bearer sk-local" -H "Content-Type: application/json" \
  -d '{"model": "Qwable-v2-oQ4e-DWQ-MTP-Vision-MLX",
       "messages": [{"role": "user", "content": "Prove there are infinitely many primes."}],
       "max_tokens": 8000, "temperature": 0.6}'

The distilled backbone also loads directly in stock mlx-lm as a qwen3_5_moe text model. MTP speculative decoding needs an MTP-aware runtime — oMLX (tested).

Recommended sampling

temperature 0.6, top_p 0.95, top_k 20. For hard reasoning / long agent runs set max_tokens ≥ 32000 — the model thinks in explicit <think>…</think> blocks. For Claude-Code-style tool use, provide an agent system prompt so it emits <tool_use> XML.

Intended use & limitations

• Built for agentic coding + hard reasoning: tool-use loops, SWE-style edits, competition math, STEM, multi-step logic.
• Reasoning/agency ≠ knowledge. Quantization (and the base distillation) transfer how to reason and act, not new facts.
• Quantization loss: 4-bit is lossy vs bf16; the fidelity ladder above quantifies it (small). For maximum quality use the bf16 base or an 8-bit quant.
• Distillation provenance: the base's training traces were generated with Anthropic's Claude Opus 4.7 / Fable-5. Downstream users should confirm compliance with Anthropic's usage policy.

Datasets

Inherited from the base model (used for its training and for this quant's calibration — no new knowledge is introduced; calibration only aligns the 4-bit scales to the model's own outputs):

• lordx64/fable-sft-combined-v2 — Fable-5 agentic / tool-use SFT (combined)
• lordx64/agentic-distill-fable-5-sft
• lordx64/fable-tool-use-sft
• lordx64/reasoning-distill-opus-4-7-max-sft — Opus-4.7 reasoning traces

Acknowledgements

• lordx64 — the Qwable-v2 base model this repo quantizes. All capability is theirs.
• Qwen team — Qwen3.6-35B-A3B.
• Anthropic — Claude Opus 4.7 and Fable-5, the reasoning/agentic teachers for the base.
• Apple MLX — mlx, mlx-lm (DWQ · mlx_lm.quant.dwq), mlx-vlm.
• oMLX — the oq/oQe imatrix quantizer, Lightning-MTP + vision serving runtime, and OpenAI-compatible API.

License

AGPL-3.0, inherited from lordx64/Qwable-v2 (the base's Fable-5 datasets are AGPL-3.0). If you serve this model over a network, the AGPL's network-use clause applies — make your corresponding source available accordingly.

Citation

@misc{qwable_v2_2026, title={Qwable-v2}, author={lordx64}, year={2026},
  howpublished={\url{https://huggingface.co/lordx64/Qwable-v2}} }
@misc{qwen36_a3b_2026, title={Qwen3.6-35B-A3B}, author={Qwen Team}, year={2026},
  howpublished={\url{https://huggingface.co/Qwen/Qwen3.6-35B-A3B}} }