tiny-qwen3-moe-fp8-per-block

tiny-qwen3-moe-fp8-per-block

A tiny Qwen3-MoE with its routed experts quantized to per-block (128x128) FP8 in AMD Quark format. It exists to give vLLM's CI a checkpoint that exercises QuarkW8A8Fp8MoEMethod's per-block path, which no public model covered before.

The weights are random. This is a test fixture derived from amd-quark/tiny-random-qwen3_moe, not a usable language model. Generations are gibberish by construction.

What is quantized

Attention is left unquantized on purpose: k_proj and v_proj are 64 rows wide, and once vLLM merges them into a single QKV GEMM the shards cannot be split on a 128-row block grid.

Scales are stored as <proj>.weight_scale (not weight_scale_inv), matching what Quark's HF export writes for fused MoE.

Shapes

With hidden_size=128, moe_intermediate_size=256, 4 experts and 2 layers, vLLM builds the fused parameters as:

TP=1    w13_weight (4, 512, 128)   w13_weight_scale (4, 4, 1)
        w2_weight  (4, 128, 256)   w2_weight_scale  (4, 1, 2)

TP=2    w13_weight (4, 256, 128)   w13_weight_scale (4, 2, 1)
        w2_weight  (4, 128, 128)   w2_weight_scale  (4, 1, 1)

Verification

On MI350 (gfx950) with the TRITON FP8 MoE backend:

• Loads at TP=1 and TP=2, binding QuarkW8A8Fp8MoEMethod with weight_qscheme=per_block and block_shape=[128, 128].
• Dequantizing the loaded fused weights with their block scales reproduces the source checkpoint to a worst-case relative error of 0.0266, which is the FP8 block round-trip error alone. A transposed scale grid or a mis-sharded scale lands near 1.0 instead.

Usage

from vllm import LLM

llm = LLM("Adamji/tiny-qwen3-moe-fp8-per-block", enforce_eager=True)