Ornith-1.5-35B-A3B-DFlash-SGLang

Ornith-1.5-35B-A3B with DFlash in SGLang

This repository contains no model weights. It documents a recipe for serving Ornith-1.5-35B-A3B with the Qwen3.5-35B-A3B-DFlash draft model in SGLang.

Results

Tested on two A100 80GB with single-stream greedy decoding and 640-token generations. GSM8K answers were checked numerically. HumanEval solutions were run against their unit tests.

The speedup is smaller than the dense 9B model sees (about 4.6x) because this sparse MoE only activates about 3B parameters per token, so its baseline is already fast (202 vs 45 tok/s). The draft still gets 6 to 8 tokens accepted per step, which is where the 2.3x comes from, and code speeds up more than math (about 2.7x versus 2.1x). The task pass rates stayed close to the baseline in this sample. The benchmark does not claim identical text across separate server runs.

Use lmsysorg/sglang:v0.5.17-cu129. Keep transformers at 5.12.1 with SGLang 0.5.17. The bf16 weights are about 73 GB, so how many GPUs you need depends on how much VRAM you have.

On two 80 GB GPUs

This is the tested setup. The weights plus the DFlash recurrent-state cache do not fit on a single 80 GB card, so split across two:

python -m sglang.launch_server \
  --model-path ornith-ai/Ornith-1.5-35B-A3B \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/Qwen3.5-35B-A3B-DFlash \
  --speculative-dflash-block-size 16 \
  --reasoning-parser qwen3 \
  --tp 2 \
  --mem-fraction-static 0.80 \
  --max-total-tokens 40960 \
  --max-running-requests 4 \
  --mm-feature-transport cpu \
  --trust-remote-code

On one GPU with 96 GB or more

If you have a single card with at least about 90 GB (a 96 GB RTX 6000 Pro, an H200, and so on), drop the tensor parallelism and run it on one GPU. The 73 GB of weights leave enough room for the draft, the state cache, and the CUDA graphs:

python -m sglang.launch_server \
  --model-path ornith-ai/Ornith-1.5-35B-A3B \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/Qwen3.5-35B-A3B-DFlash \
  --speculative-dflash-block-size 16 \
  --reasoning-parser qwen3 \
  --tp 1 \
  --mem-fraction-static 0.85 \
  --max-total-tokens 40960 \
  --max-running-requests 4 \
  --mm-feature-transport cpu \
  --trust-remote-code

A single 80 GB card is not enough for the bf16 weights: at ~73 GB they leave too little for the state cache and graphs, which is where an early attempt failed. For a single smaller card you would need an FP8 checkpoint on a Hopper GPU (the FP8 MoE kernel needs sm90+, so it will not run on an A100), or just use the two GPU setup above.

The last four flags in both commands matter on this model. --max-total-tokens and --max-running-requests keep the KV pool and the mamba state cache small enough that the CUDA graphs still fit; without them the draft graph capture runs out of memory. --mm-feature-transport cpu avoids the pidfd_getfd syscall that some hosts block for the vision feature transport (this run sends only text).

Full reproduction files are in the GitHub repository.

Attribution

• Base model: Ornith-1.5-35B-A3B
• Draft model: Qwen3.5-35B-A3B-DFlash
• Serving engine: SGLang

This recipe and benchmark were prepared by ultimatechris. The models keep their own licenses.