Ornith-1.5-35B-A3B-BigBang-MTP-NInfer

Ornith-1.5-35B-A3B-BigBang-MTP for NInfer-4090

This repository contains a single-file, mixed-precision .ninfer conversion of EryriLabs/Ornith-1.5-35B-A3B-BigBang-MTP, built for UDPSendToFailed/ninfer-4090 on a 24 GB RTX 4090 (sm_89).

The artifact contains the target model, its working MTP head, the Vision tower, an optimized proposal head, and the Qwen3.6 DFlash draft weights. DFlash bytes are part of the self-contained file; they are loaded only when NInfer starts with --spec dflash. The artifact is not a Safetensors or GGUF checkpoint and is not intended for Transformers, llama.cpp, or unrelated NInfer forks.

Quick facts

The conversion report records NInfer source revision e4233d6. Its OpenAI cached-token reporting change was merged into the upstream RTX 4090 branch in UDPSendToFailed/ninfer-4090#5.

Model provenance

The direct upstream is a TIES merge of:

• ornith-ai/Ornith-1.5-35B-A3B, an agentic-coding post-train;
• endless-frontier/BigBang-v1, a general-purpose tune; and
• Qwen/Qwen3.6-35B-A3B, the shared base.

EryriLabs also replaced stock Ornith's untrained/random MTP tensors with the trained Qwen3.6 MTP head. See the upstream model card for the merge recipe and its original llama.cpp evaluation.

This NInfer conversion preserves that MTP component and additionally embeds the Apache-2.0 Qwen3.6-35B-A3B-DFlash checkpoint from Z-Lab. The drafter targets the shared Qwen3.6 architecture rather than this specific Ornith/BigBang merge. NInfer still verifies every proposal with the target model; the pairing worked in the smoke test below, but acceptance and speed remain workload-dependent.

Frontend compatibility note

NInfer validates tokenizer, processor, and chat-template resources against its registered Qwen3.6 frontend. The direct upstream's chat_template.jinja had a different hash and was therefore not embedded. This artifact uses the official NInfer-compatible Qwen3.6 chat template (SHA-256 e84f32a23fdda27689f868aa4a1a5621f41133e51a48d7f3efcbea2839574259) while retaining the upstream model weights.

The Qwen3.6 product name coexists with Qwen3_5MoeForConditionalGeneration / qwen3_5_moe identifiers in the upstream config. That naming is expected and is not a conversion error.

What is inside the .ninfer bundle

The converter validated 1,045 BF16 target tensors across 15 source shards and 69 BF16 DFlash tensors before writing 934 runtime tensors plus six embedded frontend resources (tokenizer.json, tokenizer and generation configs, chat_template.jinja, and image/video processor configs). The result is one self-contained, registered runtime image.

Q4G64_F16S, for example, means 4-bit weights quantized in groups of 64 with FP16 scales; Q5/Q6 use the same group size, while W8G32_F16S uses 8-bit weights in groups of 32.

This is a quantized deployment artifact, not a lossless copy of the BF16 checkpoint. NInfer materializes only the components selected at startup: the verified runs reported 19.59 GiB resident weights for baseline text, 19.85 GiB with Vision, 20.11 GiB with DFlash, and 20.56 GiB with MTP. The 21.22 GiB file size is therefore not the same thing as per-mode VRAM usage; KV state, workspaces, CUDA Graphs, and concurrency add further runtime memory.

The published Ornith-1.5-35B-A3B-BigBang-MTP.ninfer.conversion.json is a path-sanitized conversion report with the complete tensor manifest and environment summary.

Download

Install the Hugging Face CLI, then download the repository:

hf download pyros-vault/Ornith-1.5-35B-A3B-BigBang-MTP-NInfer `
  --local-dir .\Ornith-1.5-35B-A3B-BigBang-MTP-NInfer

Verify the artifact on Windows:

Get-FileHash `
  '.\Ornith-1.5-35B-A3B-BigBang-MTP-NInfer\Ornith-1.5-35B-A3B-BigBang-MTP.ninfer' `
  -Algorithm SHA256

Serving with the intended NInfer runtime

Build a current native sm_89 version of UDPSendToFailed/ninfer-4090. Release binaries or other NInfer variants that predate the qwen3_6_35b_a3b-v2 target may not load this artifact.

The following examples use the same conservative memory profile as the CLI verification on a single 24 GB RTX 4090: a 4,096-token KV capacity with CUDA Graphs disabled. Sampling and thinking behavior remain request-configurable.

Text with MTP4

.\ninfer-serve.exe '.\Ornith-1.5-35B-A3B-BigBang-MTP.ninfer' `
  --kv-dtype rk4v4-e8 `
  --spec mtp --draft-tokens 4 --lm-head-draft `
  --max-context 4096 --kv-capacity 4096 --prefill-chunk 512 `
  --preserve-thinking --no-cuda-graph

Text with DFlash7

.\ninfer-serve.exe '.\Ornith-1.5-35B-A3B-BigBang-MTP.ninfer' `
  --kv-dtype rk4v4-e8 `
  --spec dflash --draft-tokens 7 --lm-head-draft `
  --max-context 4096 --kv-capacity 4096 --prefill-chunk 512 `
  --preserve-thinking --no-cuda-graph

Vision with MTP4

.\ninfer-serve.exe '.\Ornith-1.5-35B-A3B-BigBang-MTP.ninfer' `
  --vision --vision-max-tokens 1024 `
  --kv-dtype rk4v4-e8 `
  --spec mtp --draft-tokens 4 --lm-head-draft `
  --max-context 4096 --kv-capacity 4096 --prefill-chunk 512 `
  --preserve-thinking --no-cuda-graph

The server exposes OpenAI-compatible /v1/chat/completions and /v1/responses endpoints plus Anthropic-compatible /v1/messages at http://127.0.0.1:8080. Compatible-prefix reuse is enabled by default, and current upstream builds report reused prompt tokens as usage.prompt_tokens_details.cached_tokens.

Local verification results

These are bounded smoke/integration tests, not a benchmark suite. Tests used greedy decoding, thinking disabled, rk4v4-e8 KV, --max-context 4096, --kv-capacity 4096, --prefill-chunk 512, and --no-cuda-graph on one RTX 4090. The prompts and output lengths are not identical across every row, so the decode rates should not be treated as controlled speedup claims.

The pinned NInfer fixture (image, message JSON) contains the title NIFER VISION 731, three red circles, and a blue square to the left of a green triangle. Vision preprocessing took 0.095 seconds and the expanded prompt contained 428 tokens.

An additional ninfer-serve OpenAI smoke test returned exact SERVE_OK responses. Repeating the same 22-token prompt reported 18 cached tokens (81.8%) through usage.prompt_tokens_details.cached_tokens.

Important limitations

• Exact runtime matters: .ninfer is a registered, model-bound deployment format, not a generic container. Use the linked NInfer-4090 source tree with the Qwen3.6 35B-A3B target.
• DFlash is text-only in NInfer: --spec dflash cannot be combined with --vision. Use MTP for multimodal speculative decoding.
• Startup capabilities are fixed: the bundle contains all components, but requests cannot enable MTP, DFlash, or Vision weights that the server did not load at startup.
• Only the documented 4k profile was tested here: larger contexts, higher concurrency, and CUDA Graphs require their own VRAM and stability validation.
• Vision defaults are large: NInfer defaults to 8,192 Vision scratch tokens. Start with --vision-max-tokens 1024 on a 24 GB card and increase only when the workload needs it.
• No hosted inference: the Hugging Face Inference API cannot execute .ninfer files.

Conversion command

The artifact was produced with the dedicated NInfer converter:

.\.venv\Scripts\python.exe -m tools.convert.qwen3_6_35b_a3b.convert `
  --model '..\models\Ornith-1.5-35B-A3B-BigBang-MTP' `
  --dflash-model '..\models\Qwen3.6-35B-A3B-DFlash' `
  --out '..\models\Ornith-1.5-35B-A3B-BigBang-MTP.ninfer' `
  --device cuda

Conversion completed in 110.54 seconds on the RTX 4090 after strict config, tensor-inventory, dtype, and frontend-resource validation.

License and credits

The direct EryriLabs model is published as MIT and attributes Ornith, BigBang, and Qwen. The embedded Z-Lab DFlash component is Apache-2.0. See LICENSES.md for component-level attribution and source links. NInfer runtime code is not redistributed in this repository; obtain it from its own Apache-2.0-licensed upstream repository.

Converted, tested, and packaged by pyros-vault.