Solar-Open2-250B-NVFP4A16

Solar-Open2-250B-NVFP4A16

NVFP4A16 (W4A16: FP4 weights, BF16 activations; compressed-tensors) quantization of upstage/Solar-Open2-250B (250B-A15B MoE, 320+1 experts, KDA linear attention, 1M context). Quantized by Lna-Lab (@Tono_Ken3) on 12x RTX PRO 2000 Blackwell / TR PRO 9985WX, llm-compressor 0.12 + upstage transformers fork (v5.14.1-solar-open2).

Recipe notes

Our focus here is a fully documented conversion: every trap we hit is written down.

• Fused-expert unfuse: SolarOpen2Experts stores all 320 experts as fused 3D tensors, invisible to llm-compressor's targets="Linear". We unfuse into per-expert mlp.experts.{e}.{gate,up,down}_proj Linears (DeepSeek-V3 naming convention) with a functionally identical routing forward, then quantize.
• Quantized: all expert Linears, shared_experts, attention q/k/v/o_proj.
• Kept BF16: lm_head, router (mlp.gate), KDA small projections (b/f_a/f_b/g_a/g_b/g_proj) and q/k/v conv1d.
• Calibration: 48 samples x 2048 (neuralmagic/calibration), CPU eager (data-dependent MoE routing is not fx-traceable; basic pipeline).
• Sanity: post-unfuse forward answered "The capital of Korea is" -> " Seoul" before calibration.

Serving

Expert weights use per-expert (unfused) names. Serve with the upstage vLLM fork; a fused-name loader may require a weight-name mapping shim (notes to follow).

MEASURED: TP=4 x PP=3 on 12x RTX PRO 2000 (16GB, 60W cap): single-stream 64.9 t/s, C8 aggregate 217.8 t/s. Probes: exact answers on number-theory and AIME-style tasks.

License

Upstage Solar License (Apache-2.0-based; commercial use permitted). Per Section 4(e), this derivative keeps the "Solar" brand in its name.

Why W4A16 (a recipe-specific observation)

Our first bake used full NVFP4 W4A4 (48x2048 CPU calibration, basic pipeline). That build loaded and ran at full speed but emitted only empty/special tokens; requantizing weights-only (NVFP4A16) with the identical calibration was fully stable. We report this as an observation under OUR calibration recipe, not a property of the architecture — W4A4 may well be achievable with different calibration choices.