Qwen3.5-35B-A3B-sabrewing-int4

Qwen3.5-35B-A3B — sabrewing int4 container

An int4 conversion of the text tower of Qwen/Qwen3.5-35B-A3B, packaged in the container format used by sabrewing, a dependency-free C runtime for MoE inference.

71.9 GB bf16 → 21.7 GB int4.

Read this first

• No released engine can load this. The format is sabrewing's own per-layer container, not GGUF, not AWQ/GPTQ, not a transformers-loadable checkpoint. sabrewing's Qwen3.5 engine (qwen35.c) does not exist yet — only the linear-attention kernel does. If you are looking for something you can run today, you want the original bf16 weights or a GGUF build.
• Not validated end-to-end. The quantization is verified bit-exact against its own reference quantizer, and the gated-delta-net kernel is validated against transformers (relative L2 3e-7) — but no forward pass of the whole model has ever been run from these weights, so there is no token-exact oracle result and no perplexity number. Treat this as a build artifact, not a validated release.
• Text only. Qwen3.5-35B-A3B is multimodal; the 333 model.visual.* tensors are not included. Image and video input are not representable in this container.

Why this model

Qwen3.5-35B-A3B replaces softmax attention with a gated delta net (recurrent linear attention) in 30 of its 40 layers — only every 4th layer is full attention. The per-sequence state is one [128 × 128] matrix per head per layer, 2.1 MB per layer independent of context length, against 134 MB per full-attention layer at 32k. Across a 32k conversation that is 67 MB of recurrent state plus 1.34 GB of KV, where an all-softmax model of the same shape would need 5.37 GB.

Combined with int4 experts, the whole 21.7 GB container fits in a single 48 GB GPU with room for many concurrent sequences — which is the workload sabrewing targets (long-running agents, aggregate throughput over peak single-stream).

There is also an MTP draft head in here (out-mtp.safetensors, one full-attention decoder layer with its own 256-expert MoE), which is the interesting part for speculative decoding: at batch 1, decode is bound by reading every active weight once per token, and speculation is the only lever that produces more than one token per weight-read.

Layout

Names are normalized: the multimodal checkpoint's model.language_model. prefix is stripped to model., so layers are model.layers.N.*.

Precision policy

• int4 — routed experts, >90% of parameters. Per-output-row f32 scale, nibble-8 packing (low nibble = even column, high nibble = odd, offset +8):

  model.layers.N.mlp.experts.gate_up_proj      u8 [E*2I, D/2]   + .qs  f32 [E*2I]
  model.layers.N.mlp.experts.down_proj         u8 [E*D,  I/2]   + .qs  f32 [E*D]
  

  gate_up is block-concatenated per expert (all gate rows, then all up rows), i.e. exactly F.linear(x, gate_up_proj[e]).chunk(2, dim=-1). E=256, I=512, D=2048.
• f32 — every norm, the router (mlp.gate.weight), A_log, dt_bias, conv1d.weight, shared_expert_gate.weight. Small, and each feeds a nonlinearity or a narrow reduction where a bf16 rounding is not averaged away.
• bf16 passthrough — everything else: linear_attn and self_attn projections, shared expert, embeddings, lm_head.

linear_attn.conv1d.weight is squeezed from Conv1d's [C, 1, 4] to [C, 4].

Max row-relative quantization error on the routed experts is 7.14%, which is the int4 per-row bound (1/14 of the row maximum) — expected, not a defect.

Reproducing

python3 c/tools/convert_qwen35_int4.py \
  --indir  /path/to/Qwen3.5-35B-A3B \
  --outdir /path/to/qwen35_i4
python3 c/tools/convert_qwen35_int4.py --selftest    # quantizer + expert-layout tests

The converter is in the sabrewing repo. --xbits 8 produces int8 experts instead; --vision additionally emits the vision tower as out-visual.safetensors (bf16, still unsupported).

Credit

All model weights are Qwen's; this repository only requantizes and repacks them. The original model, its architecture, and its training are the work of the Qwen team — see Qwen/Qwen3.5-35B-A3B. Apache-2.0, inherited from the base model.