Qwen3-30B-A3B-RFID60-Linear-Factors

Qwen3-30B-A3B RFID 60% - Linear (identity activation)

This is a factor-only research checkpoint. It stores the original rank-heterogeneous RFID representation rather than materialized dense expert weights. Unchanged parameters are inherited from Qwen/Qwen3-30B-A3B-Instruct-2507.

Only expert gate_proj and up_proj matrices are compressed. down_proj and all non-expert parameters remain in the base model. The checkpoint uses 32 groups of four experts, heterogeneous group ranks, basis blending, and the 3% RFID sparse-projection residual.

Format

Each file in factors/ contains one layer/projection family in the lossless rfid-ragged-factors-v1 format. Zero padding and derivable rank masks from the training checkpoints are omitted. All learned floating-point tensors remain FP32; projection indices are losslessly stored as int32. This conversion does not quantize or materialize the factors.

The included rfid_factor_checkpoint.py supports per-expert materialization and direct two-GEMM execution of the factorized main path. Vanilla Transformers and upstream vLLM do not natively execute this checkpoint.

Evaluation

Zero-shot evaluation uses normalized accuracy for OpenBookQA, ARC-e, HellaSwag, and PIQA, and raw accuracy for WinoGrande and MathQA.

See evaluation/ for the complete lm-eval and perplexity JSON artifacts.