PAW-35B-A3B-GGUF

PAW-35B-A3B

If this model is useful to you, support the work by following @nawariokra on X, or buy me a coffee on Ko-fi.

<a href="https://ko-fi.com/F4P3296XP7"><img src="https://ko-fi.com/img/githubbutton_sm.svg" alt="Support me on Ko-fi" height="32"></a>

A 1.5415 bpw trellis-coded quantization of Qwen/Qwen3.6-35B-A3B that runs on one 24 GB GPU at 133–183 tok/s.

Requirements — read this first

This is not a standard GGUF. It uses the PAW trellis format (paw arch) and requires the llama-paw fork. Stock llama.cpp cannot read this model.

Honest summary

PAW is at parity on reasoning and about 6% down on code. If you generate code, that cost is real and statistically significant (see SCORECARD.md). If you reason, it is close to free.

Every number in the scorecard was measured by us on one RTX 3090 with byte-identical harnesses against our own Q8_0 measurement of the parent — nothing copied from model cards.

Serving

See SERVING.md for the tuned command (+35% over naive serving, byte-identical output). Short version:

• requires llama-paw, not stock llama.cpp
• speculative decoding with a dflash drafter (--spec-type draft-dflash)
• enable_thinking:false is required — see the loop limitation
• KV at q8_0 saves 966 MiB at 128k with no measurable quality cost

Limitations

• Code generation retains ~93–96% of parent depending on task; paired McNemar confirms the deficit is real, not noise (HumanEval+ p=0.035, MBPP+ p=0.008).
• No-think mode is required; thinking-mode behavior is unmeasured.
• Long-context quality above ~28k tokens is unmeasured.
• Quantized KV needs a build with flash attention enabled.

Integrity

sha256 c3d999f845ca33120764262527bfb2c284acd9006ba827525df0bfe5d4c6f218  PAW-35B-A3B-paw.gguf

License

Apache 2.0, inherited from the base model Qwen/Qwen3.6-35B-A3B.