Ornith-1.0-9B-NVFP4-ModelOpt

Ornith-1.0-9B NVFP4 (ModelOpt)

NVFP4 (W4A4) quantization of deepreinforce-ai/Ornith-1.0-9B, produced with the NVIDIA TensorRT Model Optimizer (ModelOpt). This is a derivative checkpoint, not the original model; see the base model card for the model's capabilities, training, and general usage.

Recipe note: this is the uniform W4A4 variant, which also quantizes the Gated-DeltaNet linear-attention path. With the required serving fixes applied (see below) it is coherent and scores GSM8K 94.0% (see Benchmarks). Some other NVFP4 builds keep the linear-attention path in BF16; this one does not.

Serving (SGLang)

Serving this checkpoint correctly on SGLang needs two fixes, both applied at launch by the serving launch script:

1. The NVFP4 scalar-scale fix, sgl-project/sglang#29151 (merged to main, NOT in 0.5.15). Without it the fused Gated-DeltaNet in_proj_qkv NVFP4 scale loads into only one slot, corrupting the global alpha, and the model emits word-salad. This is the critical one for this build.
2. The quantized-attention loader changes in qwen3_5.py, sgl-project/sglang#31220: enable quantized attention for modelopt_fp4 + load the baked FP8 KV-cache scales.

Serve on Blackwell (e.g. DGX Spark GB10 / sm121) with:

--quantization modelopt_fp4 \
--attention-backend flashinfer \
--fp4-gemm-backend flashinfer_cutlass \
--kv-cache-dtype fp8_e4m3 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder

See the Notes section below for the exact patch details.

<!-- BEGIN quant:preview -->

Quantization details (auto-generated)

• source model: deepreinforce-ai/Ornith-1.0-9B
• qformat: nvfp4 kv_cache: fp8
• calibration: 512 samples from each of pg19, cnn_dailymail
• producer: NVIDIA ModelOpt 0.45.0
• generated: 2026-07-15T10:23:44Z

Before/after sample generation was skipped for this run (SKIP_GENERATE=1).

<!-- END quant:preview --> <!-- BEGIN quant:remarks -->

Notes

Serving requires an attention-quant loader patch

This checkpoint quantizes attention as well as the FFN (uniform W4A4), unlike NVIDIA's own Qwen3.5 NVFP4 exports which keep attention in BF16. SGLang's qwen3_5.py hardcodes attention to unquantized for any modelopt_fp4 checkpoint, so it must be patched at launch time. This is applied via quantizer/sglang_launch_sm121.sh (a Python heredoc that flips the forced-None override so ModelOptFp4Config.is_layer_excluded() decides per-prefix instead). Upstreamed in sgl-project/sglang PR #31220. The patch is a no-op on NVIDIA/attention-BF16 checkpoints.

Serving requires the NVFP4 scalar-scale fix (sglang PR

This build quantizes the Gated-DeltaNet linear-attention path, whose fused in_proj_qkv loads its per-tensor NVFP4 scale (weight_scale_2 / input_scale) via a tuple shard id. SGLang before PR #29151 (commit cfc3d0555e, merged to main, NOT in 0.5.15) hardcodes shard_id=0 for that scalar in MergedColumnParallelLinear / QKVParallelLinear.weight_loader_v2, leaving the other logical slots uninitialized; the ModelOpt NVFP4 path then .max()es over them so a garbage slot becomes the runtime global alpha, producing word-salad. Serve on a SGLang that includes PR #29151, or apply it at launch (this build's launch script does). Confirmed against a BF16 baseline and a linear-attn-excluded NVFP4 build (both coherent): the failure is this scale-loading path, not the recipe.

Uniform W4A4 recipe, validated

This is the uniform W4A4 variant: the dense FFN, the full attention, AND the Gated-DeltaNet linear-attention path are all NVFP4 (FP8 KV cache). It has cleared a coherence smoke-serve (capital-of-Australia, arithmetic, code generation all correct) and GSM8K (94.0%, see Benchmarks), with the two serving fixes above applied. Unlike some other NVFP4 Qwen3.5 builds, it does quantize the linear-attention path.

Multimodal, vision tower kept BF16

Calibration is text-only, so the vision tower is excluded from quantization and stays BF16 (avoiding the amax=0 degenerate-quant failure mode). The export ships the processor configs (preprocessor_config.json etc.) needed to serve this as a proper multimodal checkpoint.

<!-- END quant:remarks --> <!-- BEGIN quant:benchmarks -->

Benchmarks

<!-- END quant:benchmarks -->