Ornith-1.0-397B-IQ2_KS-DFlash-Drafter-GGUF

DFlash Drafter for Ornith 1.0 397B A17B IQ2_KS (GGUF)

This repository contains a drafter, not a standalone language model. It does not produce useful output by itself. Load the recommended mixed precision file Ornith-DFLASH-m1-q4.gguf with Ornith 1.0 397B A17B IQ2_KS through --model-draft.

Initialized from z-lab/Qwen3.5-397B-A17B-DFlash and retrained by distillation against an IQ2_KS target on an NVIDIA GB10.

At n_max=3 the recommended file decodes at 25.65 tokens per second against 18.71 with speculation disabled, a 37.0% increase. Against the upstream drafter at matched engine configuration, the retrained weights measured +6.1% across four boots, with a prompt level 95% interval of +1.7% to +10.5% that is conditional on those boots rather than a general estimate. Both arms were FP16 so that only the weights differ. Details and raw data for both figures are below.

Usage

llama-server \
  --model    Ornith-1.0-397B-A17B-IQ2_KS-00001-of-00005.gguf \
  --model-draft Ornith-DFLASH-m1-q4.gguf \
  --spec-type dflash:n_max=3,cross_ctx=512 \
  --n-gpu-layers 99 --n-gpu-layers-draft 99 \
  --draft-params "-wgt 17" -wgt 8 \
  --flash-attn on --parallel 1

All five target shards must sit in the same directory. --model names shard one and the remaining four are loaded automatically by filename.

The two wgt flags reserve worst case graph space for the target and draft contexts. They were present in the measurements reported below, so omitting them will not reproduce those numbers. bench/serve-bench.sh carries the full configuration.

--parallel 1 is a hard runtime restriction, not a performance preference. This ik_llama.cpp build rejects multiple speculative slots at startup, because the speculative accept path indexes the batch absolutely. It does not merely serialize them.

Compatibility

Retrained against the IQ2_KS build's own output distribution rather than the bf16 model, so it is tuned to that specific quantized target. Only IQ2_KS results are reported here.

The two quantization axes are distinct. The target is IQ2_KS. The drafter files here are FP16, Q8_0, and a mixed type described below.

Files

The file named q4 is not a standard Q4 type. It contains no Q4_* tensors at all. Verified tensor composition: 26 F32, 24 Q8_0, 12 IQ3_KS, 7 IQ4_KSS. The IQ3_KS and IQ4_KSS types are specific to ik_llama.cpp and will not load in mainline llama.cpp.

Checksums

7b9f2654dfc07c0a9197da3e6414db58202a60008a24f76387a52136f302134a  bench/Ornith-DFLASH-rt-f16-rope1e7.gguf
c2d623da10ee1008a311ccd3806c4501cbe02eec9075fb44cce4819dc13c1934  Ornith-DFLASH-m1-q4.gguf
8b6bee4dc82ca1c89a782212144495e3b874465995dad8e781562d75f452f5de  Ornith-DFLASH-m1-q8_0.gguf
186b73dce2046b072a67361ddb8f09992761eed0171f2cfbb1d251fb8f943d6e  Ornith-DFLASH-m1-f16.gguf

---

Measurement 1: what the retraining bought

A preregistered A/B. The endpoint, design and analysis were fixed in writing before any of this data existed, so the reported effect is not a best-of-grid selection. It is published as bench/PREREGISTRATION-2026-08-09-retrain-ab.md. Note the chronology is self attested: no third party timestamp was obtained before the run.

Arms. Both FP16, with byte-identical engine relevant metadata (rope.freq_base = 1e7, attention.sliding_window = 4096, sliding_window_pattern = TTTTTF). They differ in weights only.

Design. Counterbalanced replicated boots in the order A, B, B, A. Two boots per arm, so between-boot variance is measured rather than assumed to be zero. The drafter is a startup time flag, so a within-boot A/B is impossible. 8 prompts, 2 repetitions per cell per boot, greedy, 256 predicted tokens, config order reversed on even repetitions.

Primary endpoint, prespecified: paired by prompt difference at n_max=3. That value was already the established operating point for this artifact before any A/B data was collected, so it was not selected from these results. For transparency it does turn out to be arm A's highest observed mean here, while arm B peaks at n_max=4.

The primary endpoint is +1.315 tokens per second, about +6.1%, with the confidence interval excluding zero. In relative terms it spans roughly +1.7% to +10.5%.

That interval is computed across the 8 prompts after averaging each arm's two boots, so it is conditional on these four boots and does not propagate boot-to-boot uncertainty. With only two boots per arm that component cannot be estimated properly; the observed spread is reported below so a reader can judge it directly.

Two checks support it:

• The n_max=0 row is compatible with zero, +0.039 with an interval spanning zero. That cell disables speculation and exercises no drafter, so a large difference there would have indicated a problem with the arms. It is a sanity check, not evidence of absence, and is deliberately not used as a correction factor for anything on the speculative path.
• Between-boot spread is small next to the effect. At n_max=3 the two boots of arm A differed by -0.093 and the two boots of arm B by +0.403, against a +1.315 signal.

n_max=2 is reported as not significant. n_max=4 and n_max=0 were secondary and exploratory. The larger n_max=4 difference is not promoted to the headline.

Raw rows: bench/results-retrain-ab-20260809.csv, 256 rows, each carrying run_id, boot_index, arm, drafter filename, drafter sha and server binary sha.

Measurement 2: what the shipped artifact delivers

Absolute throughput of Ornith-DFLASH-m1-q4.gguf, the recommended file, measured 2026-08-09 in a single server boot with n_max varied per request. 16 runs per cell, 8 prompts times 2 repetitions, ± is the sample standard deviation.

The last row is total output tokens per verification round, computed as predicted_n / (predicted_n - accepted). It includes the token the target produces itself, so accepted draft tokens per round is exactly one lower.

Speculation on against speculation off, paired by prompt within this single boot, is +6.93 tokens per second, 95% CI [5.55, 8.32], a 37.0% increase, positive on all 8 prompts with a per prompt range of +20.8% to +52.5%. Because both conditions come from one server lifetime with one binary and one artifact, this comparison carries none of the cross-session caveats attached to Measurement 1.

The shipped mixed precision file measured faster than the FP16 file used in the A/B, 25.65 against 22.94 at n_max=3, but those come from different experiments and different boots and no controlled quantization comparison was run, so this is an observation rather than an attribution. FP16 was used for the A/B only to isolate weights. This A/B does not cover the shipped quantization.

Why n_max=3

What the published rows establish directly is the shape: step time grows close to linearly in n_max while accepted tokens per round grows sublinearly, so throughput peaks and then falls.

The usual explanation, which this data does not itself prove, is that decode here is memory bound and the target is a sparse mixture of experts, so each verify position routes to a different subset of experts and speculation cannot amortize expert weight traffic the way it amortizes the dense remainder. Treat that mechanism as a hypothesis consistent with the curve rather than a result of it.

Fitting step time to the published rows in bench/results-m1q4-20260809.csv, where step time is predicted_ms / (predicted_n - accepted), gives approximately 90.9 + 14.7 x (n_max - 1) ms over K=1 to K=12. The linear form is only a local approximation. Measured minus fit, in ms: -2.9 at K=1, -1.7 at K=2, -1.0 at K=3, -0.7 at K=4, -1.1 at K=6, +16.3 at K=8, -9.9 at K=12 and -36.6 at K=15. It tracks the interior closely and breaks down at the ends, so it should not be extrapolated.

n_max=3 is the highest observed mean for this artifact on this GB10 configuration, and is the recommended setting. It is not established as a strict optimum: its paired lead over n_max=4 is only +0.414 tokens per second with a confidence interval of [-0.568, 1.395].

---

Reproducing

Published in bench/.

Prompts cover Python, Rust, SQL, JavaScript, and code review. Sampling is temperature = 0, seed = 42, max_tokens = 256. n_max is varied per request through "speculative.n_max": K against a server booted with a high startup N_MAX, which acts as the per request ceiling on this recurrent model.

Engine. ik_llama.cpp at base commit f96eaddba8bed6a9a5e628bbf6a566775c70b49c plus local patches across 21 files, +1967 / −279 lines, published as bench/engine-local-patches.diff. The base commit alone does not reproduce the measured binary. Built 2026-08-05 with -DGGML_CUDA=ON, CMAKE_CUDA_ARCHITECTURES=native which resolves to sm_121 on GB10, and -march=armv8.2-a+dotprod+i8mm+fp16. Binaries are dynamically linked, so all four objects matter:

7b7ff16cd57264905bf3b37334a29856189eb004f0cbb0b3c3cc4d04c951a576  llama-server
6ab80960540e043ebd8001162ab3401cb267a61a53b513ce13ebb1cd2fc1c089  libllama.so
1e7e686140afdd4069956414238a8e674d04136492967ba1b2cfde635cdcebca  libggml.so
3f160b1f9dc7df27142da9a0d0e1345a7cf143557f526d77a75c1e79c5e3b305  libmtmd.so

The drafter_sha16 and server_sha16 columns in the CSVs carry 16 character prefixes of these values, not full digests; the full digests are here and in the Checksums block above.

Target. paragon-of-brah/Ornith-1.0-397B-DFLASH-GGUF, the five Ornith-1.0-397B-A17B-IQ2_KS-0000N-of-00005.gguf shards, sizes 19982543552, 19806331648, 19806331648, 19994372480 and 18176416800 bytes respectively. That repository is mutable and the Hub revision was not recorded at measurement time, so the sizes above are the only pin offered; treat the target as identified but not cryptographically fixed.

Which launcher produced which measurement. Measurement 1, the A/B, was run through the published serve-bench.sh directly, so its launcher and full configuration are published. That is not the same as full reproducibility: the target repository is mutable and its revision was not recorded, so the target is pinned only by shard sizes. Measurement 2, the absolute curve, was run through run_sweep.sh, which boots via an unpublished launcher; serve-bench.sh reproduces the configuration that script resolved, but that equivalence has not been verified by re-measuring Measurement 2 through it.

Every result row carries run and configuration identifiers. Note that bench.py appends to its output file and does not fail fast on a short grid, so always point it at a fresh path.

---

Training

The method below is readable in train/train_dflash.py, which is published here, so the objective, schedule and hyperparameters can be checked against code. What cannot be checked are the inputs and the original run: the prompts, feature pack and training logs are not published. Counts and step timings from that run are therefore omitted rather than stated as unverifiable figures, and the throughput measurements above are the claims this card stands on.

• Method: distillation from the IQ2_KS target. Target features were captured from an IQ2_KS server and the DFlash head was retrained to predict the target's own argmax continuations. The target model is frozen throughout, as are the shared target embedding and output head, which are not drafter parameters. Teacher and student are different networks: the teacher is the 397B target, the student is the 6 layer draft head.
• What the feature pack contains: target hidden state vectors and target token IDs. It does not contain drafter activations; those are produced online during training.
• Objective: masked block cross entropy with position decay weights w_k = exp(-(k-1)/gamma) and block size 16, which weights early draft positions most heavily.
• Prompts: short coding tasks written for this project, covering Python, Rust, SQL, JavaScript and code review. The set is not published, so treat any characterization of its contents as author reported.
• Anchors: the trainer samples only anchors possessing a complete context ring within a single capture segment, and each anchor additionally needs the block width in rows after it to supply targets, so anchor yield per captured row is low.
• Optimizer: AdamW with warmup then cosine decay, gradient clipping, and trainable parameters held in fp32 under bf16 autocast. Note the schedule multiplies warmup by a cosine that decays from step zero rather than applying cosine after warmup, so the nominal peak learning rate is never actually reached. The exact schedule is in train/train_dflash.py and can be read there.
• Initialization: upstream z-lab bf16 weights. Saved weights are bf16, excluding rotary and inv_freq buffers.

Checkpoint selection

The best checkpoint was chosen by an offline acceptance metric computed on a fixed anchor subset taken from a capture separate from the training pack. Because that same subset both guided selection and produced the resulting score, the score is selection validation rather than an independent estimate, and neither it nor the per anchor data behind it is published here. The selection procedure is readable in train/train_dflash.py; the throughput measurements above are the claims this card stands on.

---

Conversion requirement

The Hugging Face config for a DFlash drafter carries rope_theta nested inside rope_parameters. convert_hf_to_gguf.py reads the top level key. A straight conversion therefore writes the wrong rope frequency, and the model loads, runs, and quietly drafts badly with no error message.

The config used for these builds carries rope_theta: 10000000.0 at the top level in addition to the nested rope_parameters block. If you convert your own drafter and acceptance looks poor, check this first. Note this config was prepared separately for conversion; the trainer's in memory adjustment of rope_theta does not reach any file, because checkpoint saving writes only weights and the best checkpoint record.

---

Provenance and license

Apache-2.0, inherited from the upstream drafter z-lab/Qwen3.5-397B-A17B-DFlash, a joint Z-Lab, Modal and SGLang release. See the paper and code. The target model is deepreinforce-ai/Ornith-1.0-397B, itself derived from Qwen/Qwen3.5-397B-A17B. This repository contains a derivative of the upstream drafter weights, retrained as described above.

Reproducibility of training. The trainer is published in train/ as train_dflash.py with its featpack.py reader, verified against the compiled artifact of the original run. It is unmodified, so it still contains a local sys.path insert that you will need to adjust for your environment.

Two dependencies are not included. dflash.py, which provides DFlashDraftModel, is upstream z-lab code and should be taken from z-lab/dflash; the copy used here has sha256 80d58268f8839a22cad42516cc95d127b79fca194d25ee7a8cc4d610bc8f560e. The feature packs and the target embedding and output head dumps are omitted because of their size, several gigabytes each, so the training run cannot be reproduced from this repository alone.

Not affiliated with Z-Lab, Modal, SGLang, Qwen, or deepreinforce-ai.