Qwen3.6-35B-A3B-FP8

Qwen3.6-35B-A3B-FP8

FP8 (E4M3) quantization of Qwen/Qwen3.6-35B-A3B, checked for Thai text integrity. 67 GB → 36 GB.

Quantized MoE models can degrade low-resource languages before English shows any damage. In Thai — an abugida, where vowels and tone marks are combining codepoints — that appears as detached or misordered marks. This build was verified against that specific failure mode.

Purpose

Run Qwen3.6-35B-A3B on 1× H100 80GB, or 2× RTX 4090 24GB, without losing Thai quality.

Calibration

None — this build is data-free. FP8 derives weight scales from weight max and computes activation scales per-token at runtime, so no calibration corpus is involved and none influenced these weights. Licence is therefore Apache-2.0, inherited from the base model, with no third-party data encumbrance.

Testing

Scored against the BF16 original on identical prompts, using structural metrics over the Thai Unicode block (U+0E00–U+0E7F): orphan combining marks, marks opening a word, doubled tone marks, tone-before-vowel ordering, orphan leading vowels (เ แ โ ใ ไ), U+FFFD, lone surrogates.

Structural defects, lower is better:

No Thai degradation relative to the BF16 original. Manually confirmed: correct thanthakhat (สิทธิ์ ศักดิ์ องค์), full Thai numeral range ๐–๙ (byte-fallback in this tokenizer, the most fragile decode path), correct tone marks and combining vowels.

A combining mark with no Thai character on either side is counted as a citation, not a defect — when asked to explain Thai orthography the model correctly writes marks in isolation, e.g. เครื่องหมาย (์). Without that exemption the checker punishes correct pedagogical text and, worse, scores a healthy model as more broken simply because it explained more characters. BF16 emitted 3 such citations here, FP8 one.

Long-context test — 8,685-token retrieval-style Thai prompt: 0/24 corrupted greedy, 0/24 sampled (t=0.7, top_p=0.8), 0/24 short-context control. BF16 scores identically.

Base-model behaviour at default sampling

At this model's own generation_config defaults (temperature 1.0, top_k 20, top_p 0.95), Thai responses contain a duplicated combining mark in roughly 3% of generations — and this is inherited from the base model, not introduced by quantization. 90 generations per model, identical prompts:

At temperature 0.7, top_p 0.8 all three score 0/24. If you need Thai output to be structurally reliable, lower the temperature or mask repeated combining marks with a logits processor — do not expect a different quantization to help.

Usage

MODEL=<ORG>/Qwen3.6-35B-A3B-FP8      # or a local path to this checkpoint

# 1× H100 80GB
vllm serve "$MODEL" --quantization compressed-tensors --max-model-len 40960

# 2× RTX 4090 24GB
vllm serve "$MODEL" --quantization compressed-tensors \
  --tensor-parallel-size 2 --max-model-len 16384

from vllm import LLM

llm = LLM(model="<ORG>/Qwen3.6-35B-A3B-FP8", quantization="compressed-tensors", dtype="bfloat16")

Needs Ada (SM89) or Hopper (SM90). On Ampere (A100) there are no FP8 tensor cores — vLLM dequantizes to BF16, giving quantization error with no speedup; use INT8 W8A8 there. Do not set --kv-cache-dtype fp8 — it corrupts hybrid GatedDeltaNet models (vllm#37554).

Details

FP8_BLOCK scheme via llmcompressor 0.11.0 — E4M3, block-[128,128] weights, dynamic per-token activations, compressed-tensors / float-quantized format.

Left in BF16: lm_head, embed_tokens, norms, mlp.gate (the MoE router — quantizing it perturbs expert routing across every downstream layer), shared_expert_gate, all 30 GatedDeltaNet linear_attn.* / conv1d layers, the vision tower (333 tensors), and the MTP head. The 10 full-attention layers (3, 7, 11, 15, 19, 23, 27, 31, 35, 39) have q/k/v/o_proj quantized normally.

Reproducing: oneshot on this multimodal MoE checkpoint mangles keys (llm-compressor#2568, still present in 0.11.0) — the text prefix repeats (model.language_model.language_model.language_model.…) and the vision tower nests inside the text model, so vLLM raises KeyError: '…experts.w2_weight'. This checkpoint has been repaired: a pure rename, tensor data untouched, key count preserved 1:1. Experts are stored per-expert 2-D (61,440 tensors) rather than fused 3-D — that is correct, vLLM fuses them at load.

Limitations

• Inherits base-model behaviour, including occasional non-Thai token leakage into Thai output (e.g. Chinese 追回). Present in the BF16 original; not a quantization artifact.
• Vision tower is BF16, so savings apply to the text backbone only.
• Not instruction-tuned beyond the base model.