Qwen3.6-35B-A3B-2Bit-GSQ-hybrid-GGUF

Qwen3.6-35B-A3B 2-bit GSQ — GGUF

On NVIDIA GPUs, use llama.cpp b11390 (4 October 2026) or newer. Older CUDA builds crash on prompts of certain lengths (230, 344 and 457 tokens in our tests) with "CUDA error: an illegal memory access was encountered". Other lengths work normally, and CPU-only builds are not affected. Upstream fix: #29941.

Overview

This is a GGUF conversion of ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ. It is essentially the DASLab GSQ checkpoint repackaged into stock llama.cpp types, with the small conversion errors documented below. It is not a new quantization, fine-tune, or model release. It is distributed as a text-model GGUF plus a separate BF16/F32 vision-projector GGUF.

The routed experts use Q2_0 (source: 2-bit GSQ), while attention, linear attention, shared-expert, and LM-head weights use Q8_0 (source: INT8). Remaining text-model tensors use BF16 or F32 (source: BF16) as required by llama.cpp's canonical QWEN35MOE layout. The companion projector retains the source BF16 vision tower as BF16 or exact F32 promotions. MTP weights remain omitted.

Pinned source revisions

Q2_0 is a recent llama.cpp type. Its type definition and CPU backend landed on 7 July 2026 in commit bec4772, first released as nightly build b9913 on 8 July 2026. CUDA support followed on 30 July 2026 in commit 9b2a088. Use a current llama.cpp build; older releases cannot load or run this model correctly.

The text model and projector were tested together with unmodified llama.cpp on CPU and with 20 of 41 text-model layers plus the projector offloaded to an RTX 3060. Both runs correctly read MEN WALK ON MOON from llama.cpp's newspaper test image. These are load and short-generation checks, not benchmark or perplexity results.

Data-type mappings and their effects

GSQ 2-bit routed experts → Q2_0

Before applying their block scales, the two formats represent these integer sets:

GSQ 2-bit: {-2, -1, 0, 1}
Q2_0:      {-1,  0, 1, 2}

The source GSQ code c ∈ {0, 1, 2, 3} has integer value c - 2, which is multiplied by scale s. A stock Q2_0 code q ∈ {0, 1, 2, 3} has integer value q - 1, which is multiplied by scale d. Setting d = -s and reversing the code order makes the scaled values identical:

This directly gives the assignment-preserving conversion formula:

Q2_0 code  = 3 - c
Q2_0 scale = FP16(-s)

Each source group of 128 weights becomes two consecutive Q2_0 blocks of 64 that share the mapped scale. The 2-bit assignments are rearranged directly; the weights are not dequantized and requantized. The only numerical change is from converting each BF16 scale to the FP16 scale stored by Q2_0.

INT8 weights → Q8_0

The signed INT8 codes are copied directly into Q8_0 blocks. Each source group-of-128 scale is converted to FP16 and repeated across four Q8_0 blocks of 32. The ordinary Q8_0 quantizer is not used, so the codes—including -128 and +127—are preserved. Scale conversion is the only numerical change.

Across the full audited checkpoint, this direct Q8_0 representation had lower maximum error, mean absolute error, RMSE, and relative Frobenius error than dequantizing the same weights to BF16. It also reduced the finished text-only GGUF by 1,794,048,032 bytes compared with the BF16-fallback build.

Other tensors

Embeddings, norms, routing weights, biases, and other ordinary text tensors follow llama.cpp's canonical Qwen conversion and are stored as BF16 or F32. Expert aggregation and linear-attention permutations also follow the canonical QWEN35MOE layout.

BF16 vision tower → BF16/F32 mmproj

The source contains 333 vision tensors, all BF16. llama.cpp's stock Qwen3-VL converter stores the large vision-transformer and merger matrices as BF16 and promotes biases, normalization parameters, patch embeddings, and positional embeddings to F32. The temporal-size-2 Conv3D patch embedding becomes two Conv2D tensors, producing 334 GGUF tensors in total.

All 446,571,248 projector values were compared against the source. BF16 payloads are bit-exact and BF16-to-F32 promotion is exact, so this projector conversion has zero numerical error.

Accepted conversion errors

Stock Q2_0 and Q8_0 store their block scales as FP16, whereas the source checkpoint uses scales that are not always exactly representable in FP16. The following deviations were measured and explicitly accepted.

Q2_0 routed experts

• 1,890,976 scales with the previously identified BF16 bit pattern 0x018e underflow to FP16 zero. Each scale has magnitude 5.216256181773901e-38; setting it to zero produces a maximum weight error of 1.0432512363547802e-37 for this group.
• 15 additional scales incur FP16 subnormal rounding.
• One additional scale of magnitude 2.200249582529068e-8 underflows to zero, producing a maximum weight error of 4.400499165058136e-8 for that group.
• The maximum measured GSQ-to-Q2_0 weight error is 5.960464477539063e-8.

The direct mapping was retained rather than changing assignments to hide these errors.

Q8_0 INT8-derived weights

• 7,104 nonzero scales underflow to FP16 zero, all in seven linear-attention tensors. They produce 28,416 zero-scale Q8_0 blocks, cover 909,312 weight positions, and incur a maximum weight error of 1.6699515299478662e-6.
• No scale overflow or additional scale-loss regime was found.

Despite those underflows, Q8_0 has lower whole-tensor maximum error, mean absolute error, RMSE, and relative Frobenius error than BF16 for each of the seven affected tensors.

Basic llama.cpp usage

llama-cli \
  --model Qwen3.6-35B-A3B-GSQ-hybrid.gguf \
  --mmproj mmproj-BF16.gguf \
  --image image.jpg \
  --ctx-size 4096 \
  --gpu-layers 20 \
  --prompt "Describe this image."

Omit --mmproj and --image for text-only use. Adjust context size and GPU offload for the available memory. For grounding-sensitive image tasks, llama.cpp recommends --image-min-tokens 1024. This conversion was validated against llama.cpp commit 911f6cdc8ab8a530b2bee09ee61471a6f3178eeb.

Text quality evaluation

The unmodified text-model GGUF was evaluated on a fixed 100-document corpus using exact full-vocabulary causal cross-entropy. The corpus contains 25 documents from each of four strata: general literature, encyclopedic knowledge, source code, and multilingual text (German, Spanish, French, Italian, and Portuguese). Each document was evaluated independently with at most 1,050 tokens, for a total of 104,900 predicted tokens.

The GSQ GGUF therefore has approximately 1.502 times the perplexity of the BF16 base on this corpus. Results by corpus stratum were:

The GGUF was scored with unmodified llama.cpp at commit 911f6cdc8ab8a530b2bee09ee61471a6f3178eeb; the BF16 base was scored with a native PyTorch/Transformers evaluator on the same token sequence and objective. The comparison is therefore informative but crosses inference runtimes. This is a custom language-model perplexity evaluation, not a standardized downstream task benchmark or a multimodal evaluation.

Compared with imatrix quantizations (added October 2026)

Later measurements found that llama.cpp's own importance-matrix (imatrix) quantizations of the BF16 model are closer to BF16 than this GGUF at about the same size:

• Chat: 223 held-out chat conversations scored against BF16 with llama.cpp (method in the GSQ-RCO card). Unsloth's file is lower on 204 of 223 conversations.
• Raw text: the 100-document corpus above, same build and settings.
• Routed experts alone: replacing this GGUF's Q2_0 experts with IQ2_XS (2.8% more bytes), quantized from BF16 with an imatrix from our chat data, lowered chat KL from 0.135 to 0.080 on our calibration conversations (with expert pruning in both arms). So GSQ's 2-bit experts do not beat imatrix IQ2_XS on this model. Note that the imatrix and the conversations come from the same distribution.

Coding, instruction following and speed were not compared. For chat, an imatrix-based file of similar size is likely the better choice.

Attribution

The model weights and GSQ quantization are from ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ, which is based on Qwen/Qwen3.6-35B-A3B. Please cite and credit the original model and GSQ authors when using this conversion.

The quantization method is described in the GSQ paper, with source code available in the ISTA-DASLab GSQ repository.

@article{gsq2026,
  title   = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author  = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurti{\'c}, Eldar and Kleinegger, Max and Alistarh, Dan},
  journal = {arXiv preprint arXiv:2604.18556},
  year    = {2026},
  url     = {https://arxiv.org/abs/2604.18556}
}

License

This GGUF conversion is made available under the Apache License 2.0, just like the DASLab source checkpoint and its Qwen base model.