Ornith-1.0-35B-NVFP4-ModelOpt

Ornith-1.0-35B NVFP4 (ModelOpt)

NVFP4 (W4A4) quantization of deepreinforce-ai/Ornith-1.0-35B, produced with the NVIDIA TensorRT Model Optimizer (ModelOpt). This is a derivative checkpoint, not the original model; see the base model card for the model's capabilities, training, and general usage.

Recipe note: this is the uniform W4A4 variant, which also quantizes the Gated-DeltaNet linear-attention path. That is more aggressive than NVIDIA's own Qwen3.5-NVFP4 exports (MoE-only, attention kept in BF16). It has been validated for coherent output and math reasoning (GSM8K 96.75%, see Benchmarks below), but is offered on that basis.

Serving (SGLang)

Because this checkpoint quantizes attention, a stock SGLang build needs three small loader changes in qwen3_5.py (enable quantized attention for modelopt_fp4, plus load the baked FP8 KV-cache scales). These are upstreamed as sgl-project/sglang#31220 and applied at launch. Serve on Blackwell (e.g. DGX Spark GB10 / sm121) with:

--quantization modelopt_fp4 \
--attention-backend flashinfer \
--moe-runner-backend flashinfer_cutlass \
--fp4-gemm-backend flashinfer_cutlass \
--kv-cache-dtype fp8_e4m3 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder

See the Notes section below for the exact patch details and validation.

<!-- BEGIN quant:preview -->

Quantization details (auto-generated)

• source model: deepreinforce-ai/Ornith-1.0-35B
• qformat: nvfp4 kv_cache: fp8
• calibration: 512 samples from each of pg19, cnn_dailymail
• producer: NVIDIA ModelOpt 0.45.0
• generated: 2026-07-14T19:05:05Z

Before/after sample generation was skipped for this run (SKIP_GENERATE=1).

<!-- END quant:preview --> <!-- BEGIN quant:remarks -->

Notes

Serving requires an attention-quant loader patch

This checkpoint quantizes attention as well as the MoE experts (uniform W4A4), unlike NVIDIA's own Qwen3.5 NVFP4 exports which quantize the MoE only and keep attention in BF16. SGLang's qwen3_5.py hardcodes attention to unquantized for any modelopt_fp4 checkpoint, so it must be patched at launch time. This is applied via quantizer/sglang_launch_sm121.sh (a Python heredoc that flips the forced-None override so ModelOptFp4Config.is_layer_excluded() decides per-prefix instead). The patch is a no-op on NVIDIA/MoE-only checkpoints.

Deliberately-risky recipe, not the default

This is the uniform W4A4 variant (attention AND experts quantized) of a Gated-DeltaNet hybrid-mamba MoE, kept for a quality A/B against the safe default. Prefer ornith-1.0-35b_nvfp4-mlp-only.yaml (experts-only, attention high precision, matches NVIDIA's own Qwen3.5-NVFP4 recipe shape) unless this uniform build has cleared a real smoke-serve quality check.

Validated coherent output on SGLang GB10/sm121

Smoke-served on SGLang GB10/sm121 with the attention-quant loader patch applied, and produced coherent, correct output on both a German-language prompt (capital of Australia: Canberra) and an arithmetic prompt (17 x 24 = 408). No word-salad was observed on the Gated-DeltaNet linear-attention path despite it being quantized to W4A4.

FP8 KV-cache scales load via the serving launch patch

A stock SGLang build would drop this checkpoint's baked per-layer FP8 KV scales, because qwen3_5 builds RadixAttention without a quant_config (so the KV scales fall back to 1.0). The serving launch patch (sglang_launch_sm121.sh) passes quant_config to RadixAttention and remaps/loads the baked k_proj.k_scale/v_proj.v_scale onto attn.k_scale/attn.v_scale. Verified: the runtime k_scale_float holds the calibrated per-layer values (~0.01 to 0.045), not 1.0. Upstreamed in sgl-project/sglang PR #31220.

Multimodal, vision tower kept BF16

Calibration is text-only, so the vision tower is excluded from quantization and stays BF16 (avoiding the amax=0 degenerate-quant failure mode). The export ships the processor configs (preprocessor_config.json etc.) needed to serve this as a proper multimodal checkpoint.

<!-- END quant:remarks --> <!-- BEGIN quant:benchmarks -->

Benchmarks

<!-- END quant:benchmarks -->