Downloads · 30 days
86
2% of all-time downloads
pocharlies/Ornith-1.0-35B-NVFP4-MTP-graft
Ornith-1.0-35B-NVFP4-MTP-graft is a image-text-to-text model from pocharlies. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
What this is: the agentic-coding fine-tune deepreinforce-ai/Ornith-1.0-35B (a Qwen3.6-35B-A3B MoE, 256 experts / 8 active) made fast on NVIDIA GB10, plus a fix for a chat-template defect that causes repeating tool-cal…
Downloads · 30 days
86
2% of all-time downloads
All-time downloads
4.3K
Public
Parameters
34.6B
22.9 GB on disk
Likes
3
Public
Click a slice to open those files.
.safetensors22.9 GB · 100%
How the weights are stored.
U833.6B · 97%
From the Hugging Face model README
What this is: the agentic-coding fine-tune deepreinforce-ai/Ornith-1.0-35B (a Qwen3.6-35B-A3B MoE, 256 experts / 8 active) made fast on NVIDIA GB10, plus a fix for a chat-template defect that causes repeating tool-call loops in agentic CLIs.
Three changes over upstream:
mtp.* tensors). We graft the 19 mtp.* tensors from nvidia/Qwen3.6-35B-A3B-NVFP4 (same architecture) onto Ornith. This enables multi-token prediction (self-speculative decoding, K=3) with ~100% acceptance — the draft and verify share the model, so the fine-tune's behaviour is preserved.lm_head W4. The BF16 lm_head (1.0 GB over 248,320 vocab, read every decode step) is re-quantized to 4-bit (compressed-tensors WNA16 int4 g32, served by Marlin).Qwen3.6-35B-A3B is a vision-language model, and so is this checkpoint. Until 2026-08-13
this card declared pipeline_tag: text-generation and never mentioned the vision tower —
that was wrong, and it is what this section fixes.
Quantization deliberately skips the whole tower (recipe.yaml: ignore: ['re:.*visual.*', …]),
so it survives in BF16, unquantized:
| check | result |
|---|---|
model.visual.* tensors in model.safetensors.index.json | 333 |
same tensor names as Qwen/Qwen3.6-35B-A3B | 333 / 333, no extras, none missing |
config.json architectures | Qwen3_5MoeForConditionalGeneration (identical to upstream) |
vision_config | identical to Qwen/Qwen3.6-35B-A3B |
preprocessor_config.json / processor_config.json / video_preprocessor_config.json | present, Qwen3VLProcessor |
chat_template.jinja | md5 52b6d51a… — byte-identical to Qwen's, emits <|vision_start|><|image_pad|><|vision_end|> |
So nothing in this repo blocks the vision path. We have not exercised it ourselves — every number below is text-only, on a text-only agentic-coding workload. If vLLM logs
Model … is treated as multimodal but has no registered multimodal processor; running in text-only mode.
that warning comes from vLLM's architecture resolution, not from the weights: it means the
model class vLLM resolved has no _processor_factory attached. Qwen3_5MoeForConditionalGeneration
has been in the multimodal section of vLLM's model_executor/models/registry.py since
v0.25.0, so a stock ≥0.26 wheel should resolve it. Check the Resolved architecture: line in
your startup log — if it is anything other than Qwen3_5MoeForConditionalGeneration (in
particular if you also see "has no vLLM implementation, falling back to Transformers
implementation"), the fix is on the engine side, not the checkpoint.
If you are bringing vision up, drop --speculative-config first. MTP self-speculative
decoding together with multimodal inputs is a combination we have never measured. Confirm an
image round-trips, then put MTP back.
Upstream deepreinforce-ai/Ornith-1.0-35B ships a chat_template.jinja that removed the conditional guarding <think> re-rendering for past assistant turns, while leaving the ns.last_query_index computation in place (an unfinished edit). Verified by hash: upstream Ornith's template is md5 fa5eb1ee…, whereas Qwen/Qwen3.6-35B-A3B and nvidia/Qwen3.6-35B-A3B-NVFP4 are byte-identical to each other (md5 52b6d51a…). The only difference is:
{#- canonical Qwen3.6 (correct): strip thinking from older turns #}
{%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{#- upstream Ornith: ALWAYS re-renders old <think> #}
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
Effect: every prior assistant turn's chain-of-thought is replayed into the prompt, so the model re-reads a stale plan ("I should call tool X") and re-issues it — reported as constant tool-call loops with agentic clients (e.g. qwen-code). It also bloats context. This repo ships the canonical template; if you use upstream Ornith directly, pass vLLM --chat-template with the Qwen3.6 template.
Also worth knowing: Qwen forbids greedy decoding.
temperature 0does not make tool-calling safer — it causes degradation and infinite repetition loops. For quantized checkpoints usetemperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5.
| context | this (graft, MTP K=3, lm_head-W4) | plain nvidia/Qwen3.6-35B-A3B-NVFP4 |
|---|---|---|
| 1K | 174 tok/s | ~63 (no MTP weights → no spec-decode) |
| 8K | 170 tok/s | 155 |
| 131K | 133 tok/s | ~88 |
Prefill @131K: 47.8 s (vs 518 s pre-optimization, 10.8×). TTFT cached @131K: 1.9 s. MTP acceptance ~100% at K=3. K=4 is a regression (accept rate at position 4 falls below breakeven). Tooling quality preserved: 346/370 on our internal 370-case battery (BF16 head scored 343/370 — within noise). Upstream Ornith agentic-coding scores (per its card): SWE-bench Verified 75.6, SWE-bench Pro 50.4, Terminal-Bench 2.1 64.2, Claw-eval 69.8.
vllm serve pocharlies/Ornith-1.0-35B-NVFP4-MTP-graft \
--served-model-name ornith-1.0-35b-nvfp4-mtp \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--enable-prefix-caching --enable-chunked-prefill \
--enable-auto-tool-choice --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
--default-chat-template-kwargs '{"enable_thinking":false}' \
--attention-backend flashinfer \
--max-num-seqs 4 --max-num-batched-tokens 32768 --gpu-memory-utilization 0.55 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--trust-remote-code
env: VLLM_MARLIN_USE_ATOMIC_ADD=1 VLLM_NVFP4_GEMM_BACKEND=marlin
This command is what we run for text-only agentic coding, and it is what every number on this card was measured with. It does not disable vision — but see Vision above: if you want images, bring the engine up without
--speculative-configfirst.
KV cache
nvfp4is impossible on GB10. FlashInfer requiressm100f(Blackwell datacenter); GB10 issm121→ the engine aborts. Use--kv-cache-dtype fp8.
MIT (inherited from Ornith-1.0-35B). Ornith weights © deepreinforce-ai (MIT); the 19 grafted mtp.* tensors are from nvidia/Qwen3.6-35B-A3B-NVFP4 (Apache-2.0); base architecture Qwen/Qwen3.6-35B-A3B; chat_template.jinja is the canonical Qwen3.6 template. This repo adds the MTP graft, the lm_head re-quantization, and the template fix.
You were reading the wrong repo, and it is not obvious. sakamakismile/Ornith-1.0-35B-NVFP4
publishes only model.safetensors — no model-mtp.safetensors, no
model.safetensors.index.json, no model-lmhead-w4.safetensors. Verified against the Hub
API:
| file | upstream repo | this repo |
|---|---|---|
model.safetensors | 21,901,607,200 B | 20,884,484,712 B (lm_head split out) |
model-mtp.safetensors | absent | 1,689,283,688 B |
model-lmhead-w4.safetensors | absent | 286,064,960 B |
model.safetensors.index.json | absent | 12,478,012 B |
Without the index, vLLM loads a single file, the mtp.* tensors do not exist, and you get
1 token per forward pass instead of ~3.9 — no matter how correct your flags are.
This repo is the complete, ready-to-serve checkpoint. Point at it and use the serving command above. Nothing to assemble.
assemble.py is included only for the case where you want to graft the MTP onto your own
copy of the upstream checkpoint instead of downloading this one. It reads safetensors
headers (no 22 GB load) and refuses to emit an index that would not load.
Re-measured on the same hardware with a different harness, discarding every sample taken while another request — or another pod on the same time-sliced GPU — was in flight:
| workload | tok/s | MTP tokens/step | acceptance |
|---|---|---|---|
| code edit (249-token prompt, 700 max) | 154.5 | 3.91 / 4.0 | 94-98 % |
| long context (52K prompt) | 149.4 | 3.73 | 91 % |
| free-form prose (512 tokens) | 104.5 | 2.60 | 51-60 % |
Consistent with the 1K/8K figures above; the spread is workload, not variance. Always state the workload with a tok/s number — acceptance, and therefore throughput, swings by 2× between prose and code editing on the same engine.
Two measurement traps worth repeating, both of which cost us hours:
/metrics and discard samples taken with num_requests_running > 0.curl /metrics | grep spec_decode_num_accepted_tokens_total.
We once concluded speculative decoding "gave no speedup" when a kubectl apply had
silently reverted the flag. "On but useless" and "never enabled" look identical from
throughput alone.For qwen3_moe models (e.g. Qwen3-Coder-30B-A3B) this MTP module is not graftable —
different architecture, dimensions and vocabulary. The equivalent there is an EAGLE3 draft
head; complete measured recipe at
pocharlies/Qwen3-Coder-30B-A3B-NVFP4-EAGLE3-DGXSpark.
Speed, tool-calling and agentic results for this model alongside three others on the same DGX Spark — including two optimizations that were measured and rejected — with all raw evidence:
https://huggingface.co/datasets/pocharlies/dgx-spark-moe-benchmarks