Downloads · 30 days
21
1% of all-time downloads
Ornith-1.0-397B-W4A16-AutoRound-DFlash
fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash
Ornith-1.0-397B-W4A16-AutoRound-DFlash is a text generation model from fraserprice. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
Pilcothink's AutoRound W4A16 quant of Ornith-1.0-397B, packaged with the z-lab DFlash draft for speculative decoding. Tuned on 4× RTX Pro 6000 Blackwell (SM120, 4×96GB).
All-time downloads
1.8K
Public
Repo size
210 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors210 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Model type
- qwen3_5_moe
- Access
- Public
- Created
- Jul 1, 2026
- Updated
- Jul 2, 2026
- SHA
- 01616779
Try a prompt
Base models
Model card
From the Hugging Face model README
Ornith-1.0-397B W4A16 + DFlash (vLLM, RTX Pro 6000)
Pilcothink's AutoRound W4A16 quant of Ornith-1.0-397B, packaged with the z-lab DFlash draft for speculative decoding. Tuned on 4× RTX Pro 6000 Blackwell (SM120, 4×96GB).
NOTE: This can likely be improved a lot further.
Usage
hf download fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash --local-dir Ornith-1.0-397B
cd Ornith-1.0-397B
./serve-rtx-pro.sh
OpenAI-compatible server on port 8000: TP4, 262144 context, reasoning parser qwen3, tool parser
qwen3_xml. Runs on voipmonitor/vllm:eldritch-enlightenment-v2226f26-b12x15cd38c-cu132-20260629
(vLLM 0.11.2.dev + native support for the DFlash sliding-window drafter). First run pulls the image
and downloads the draft weights. Override via PORT, TP, MODEL_DIR, DRAFT_DIR, IMAGE.
Config notes
--attention-backend FLASHINFER: split-KV decode keeps the multi-token verify step at ~20 ms per step regardless of context length.num_speculative_tokens=12: ~40% faster decode than k=8 at concurrency 1, equal at 2. k=16 (the draft's native block) is slower.- Keep the KV cache bf16 and
--async-schedulingoff; both alternatives lose speed and/or acceptance on this hybrid-GDN architecture. NCCL_P2P_LEVEL=SYS+VLLM_MARLIN_USE_ATOMIC_ADD=1: +13% decode, +40% prefill on 4× PCIe GPUs.
Benchmarks
4× RTX Pro 6000 Blackwell, TP4, streaming OpenAI requests, real coding prompts generated to natural EOS (≤2048 tokens), 3 repeats. Vanilla = same image and quant, default vLLM serving per the base model card, no speculative decoding. Accept = mean accepted tokens per verify step.
| prompt | conc | decode tok/s | vanilla tok/s | uplift | accept | TTFT p50 | ITL p50 |
|---|---|---|---|---|---|---|---|
| 1,000 | 1 | 197.7 | 121.2 | 1.63× | 4.20 | 152 ms | 21.0 ms |
| 10,000 | 1 | 139.5 | 119.9 | 1.16× | 3.71 | 1.28 s | 21.1 ms |
| 100,000 | 1 | 148.5 | 107.9 | 1.38× | 3.41 | 14.2 s | 22.6 ms |
| 1,000 | 3 | 149.1 | 80.1 | 1.86× | 4.20 | 392 ms | 29.2 ms |
| 10,000 | 3 | 104.7 | 75.6 | 1.38× | 3.45 | 2.86 s | 29.7 ms |
| 100,000 | 3 | 69.8 | 47.1 | 1.48× | 3.61 | 28.1 s | 33.6 ms |
Prefill throughput 6.7–8.7k tok/s.
Credits
- Model: DeepReinforce Team (MIT)
- Quant: Pilcothink, rehosted unmodified
- DFlash: z-lab (arXiv 2602.06036)
- vLLM image: voipmonitor eldritch builds
- Task
- Text Generation
- Library
- transformers
- Type
- qwen3_5_moe
- License
- mit
- Created
- Jul 1, 2026
- Updated
- Jul 2, 2026