Ornith-1.5-397B-MXFP4

Ornith-1.5-397B-MXFP4

MXFP4-quantized build of ornith-ai/Ornith-1.5-397B — the same model at roughly half the disk and memory footprint, targeting single-GPU serving on 288 GB AMD Instinct MI355X.

• Checkpoint size: ~239 GB
• Quantization: OCP MXFP4, block size 32
• Quantizer: AMD Quark 0.12.post1
• GPU tested: 1× AMD Instinct MI355X, 288 GB HBM
• Tensor parallelism: TP=1
• Architecture: Qwen3.5 MoE / Ornith 1.5
• Routed MoE expert weights are MXFP4; attention, shared experts, gates, MTP, visual components and lm_head remain BF16.

Recommended serving

The checkpoint is tested with SGLang ROCm 7.2.4 on MI355X. AITER and native NEXTN/MTP are supported.

docker run --rm \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add video \
  --security-opt seccomp=unconfined \
  --ipc=host \
  --shm-size=32g \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e SGLANG_USE_AITER=1 \
  -e SGLANG_USE_AITER_UNIFIED_ATTN=1 \
  -e AITER_FLYDSL_FORCE=1 \
  -e SGLANG_MAMBA_SSM_DTYPE=bfloat16 \
  lmsysorg/sglang-rocm:v0.5.18-rocm724-mi35x-20260830 \
  python3 -m sglang.launch_server \
    --model-path truespirit/Ornith-1.5-397B-MXFP4 \
    --tp 1 \
    --attention-backend aiter \
    --page-size 16 \
    --reasoning-parser deepseek-r1 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 2 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 3 \
    --mem-fraction-static 0.95 \
    --max-running-requests 4 \
    --context-length 81920 \
    --host 0.0.0.0 \
    --port 8000

The server exposes an OpenAI-compatible API.

Measured on 1× MI355X

With the configuration above:

• ~180–200 output tok/s single-stream decode
• ~290 output tok/s aggregate at concurrency 4
• ~0.1 s TTFT on the tested 16K-prompt workload
• ~86K tokens measured single-request input ceiling
• two concurrent ~62K-token inputs verified

Results are serving benchmarks for this specific hardware/runtime configuration and are not model-quality benchmarks.

For general reasoning, temperature=0.6, top_p=0.95 is a useful starting point. For coding/agent workloads, temperature=1.0, top_p=0.95 is also suitable.

Quantization

Only routed MoE expert weights (*.mlp.experts.*) were quantized to MXFP4. Components kept at higher precision follow the AMD Qwen3.5 MXFP4 recipe.

Quantization can cause numerical differences from the source FP8 checkpoint.

License

MIT, inherited from the base model. See the original ornith-ai/Ornith-1.5-397B model card for model details, intended use and limitations.