Huihui-Ornith-1.5-35B-A3B-Uncensored-NInfer-RTX5090-Guide

Huihui-Ornith-1.5-35B-A3B Uncensored @ 262K context on a single RTX 5090 — with NInfer

A complete, reproducible guide to run an uncensored 35B-A3B MoE model at its full native 262,144-token context on one consumer RTX 5090 (32 GB), using a 4-bit (K8V4) KV cache and MTP speculative decoding — reaching hundreds of tok/s on a single GPU, with a ~7,400 tok/s prefill at 185k tokens.

The uncensored base we used is huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated (an abliteration of ornith-ai/Ornith-1.5-35B-A3B). A ready-to-run .ninfer artifact is here: pcmaker/Huihui-Ornith-1.5-35B-A3B-Uncensored-NInfer.

Everything here is open source. Nothing below contains any private key, password or host.

Intended use. Uncensored / abliterated weights remove the model's built-in refusals. This guide is for authorized work only — red-teaming and security testing of your own applications, safety research, and offline experimentation. You are responsible for what you generate. Do not use it to harm people.

---

TL;DR — what we measured (RTX 5090, thinking on/off)

Quality is unchanged vs. the base model — MTP proposals are verified by the main model, so wrong drafts are discarded.

---

Why this model is different from Qwen3.8-27B

Ornith-1.5-35B-A3B uses the qwen3_5_moe architecture (same family as Qwen/Qwen3.6-35B-A3B): a sparse Mixture-of-Experts (256 experts, 8 active/token, ~3B activated), a hybrid Gated-DeltaNet + gated-attention decoder (40 layers, 10 full-attention + 30 GDN), a built-in MTP draft module, and a vision tower. NInfer serves it through its qwen3_6_35b_a3b target.

Two practical consequences vs. the dense 27B:

1. No external quantization step. NInfer's 35B-A3B converter takes BF16 + the DFlash companion and does its own mixed quantization internally (Q4/Q5/Q6/W8). You do not feed a pre-made NVFP4 checkpoint — a ready NVFP4 repo will not work here.
2. MTP beats DFlash for a heavily fine-tuned base. The model ships its own trained MTP head, so --spec mtp is self-aligned (~76-83% accept). The external z-lab DFlash was trained on the official checkpoint's residuals, so on a strongly retrained base like Ornith its accept rate is lower — use MTP.

---

Hardware & OS

• GPU: NVIDIA RTX 5090, 32 GB, Blackwell, compute capability sm_120.
• OS: Ubuntu 24.04 LTS. CUDA Toolkit 13.3 (nvcc 13.3). Blackwell + CUDA 13.x required.
• CPU/RAM: any modern desktop; we used a 12-core Ryzen with 128 GB DDR5.
• Disk: ~110 GB free during the build (BF16 base ~72 GB + DFlash ~0.8 GB + ~22.8 GB artifact).

---

Open-source components used

---

Step-by-step

0. Prerequisites (Ubuntu 24.04)

sudo apt-get update
sudo apt-get install -y cmake ninja-build gcc g++ pkg-config \
  libavformat-dev libavcodec-dev libavutil-dev libswscale-dev \
  libavdevice-dev libcurl4-openssl-dev
nvcc --version   # -> release 13.3
pip install -U "huggingface_hub[cli]" transformers safetensors torch

1. Build NInfer

git clone --depth 1 https://github.com/Neroued/ninfer.git && cd ninfer
export CUDA_HOME=/usr/local/cuda-13.3 PATH="/usr/local/cuda-13.3/bin:$PATH"
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j 6            # ~30 min
export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:$LD_LIBRARY_PATH

The qwen3_6_35b_a3b target is compiled into build/apps/ninfer-serve by default — no extra flags needed.

2. Easiest path — download the ready artifact

hf download pcmaker/Huihui-Ornith-1.5-35B-A3B-Uncensored-NInfer --local-dir ./art

Then jump to step 6 (serve). To build it yourself, steps 3-5:

3. Download the two conversion sources (BF16)

hf download huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated --local-dir src/base-bf16   # ~72 GB
hf download z-lab/Qwen3.6-35B-A3B-DFlash                     --local-dir src/dflash      # ~0.8 GB

4. Fix two format details the converter is strict about

(a) chat template. The converter hash-pins six frontend files to the official Qwen/Qwen3.6-35B-A3B. For Ornith, five of six are byte-identical (including the 12.8 MB tokenizer.json); only chat_template.jinja differs. Swap in the official one:

hf download Qwen/Qwen3.6-35B-A3B chat_template.jinja --local-dir /tmp/qwtmpl
cp /tmp/qwtmpl/chat_template.jinja src/base-bf16/chat_template.jinja

(b) fused MoE experts. This base ships the MTP layer's experts unfused (mtp.layers.0.mlp.experts.<e>.{gate,up,down}_proj.weight), while NInfer expects the stacked experts.gate_up_proj / experts.down_proj. The text layers are already fused. Run fuse_mtp_experts.py once (CPU only, rewrites a single shard):

python fuse_mtp_experts.py src/base-bf16

5. Convert to a .ninfer artifact

cd ninfer
python3 -m tools.convert.qwen3_6_35b_a3b.convert \
  --model ../src/base-bf16 --dflash-model ../src/dflash \
  --out ../models/ornith_35b.ninfer --device cuda

NInfer quantizes internally and writes ~940 objects, ~22.8 GB, in under a minute.

6. Serve it

See serve_ninfer.sh:

export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:$LD_LIBRARY_PATH
./ninfer/build/apps/ninfer-serve models/ornith_35b.ninfer \
  --host 0.0.0.0 --port 8188 --model-id ornith-35b \
  --max-context 262144 --kv-dtype k8v4 \
  --spec mtp --draft-tokens 3 --default-thinking-budget 8192

OpenAI- and Anthropic-compatible endpoints on port 8188. Useful flags: --api-key, --model-id <name>, --kv-dtype k8v4|fp8|int8|bf16. To try DFlash instead of MTP: --spec dflash --draft-tokens 7 --lm-head-draft (expect lower accept on this base).

---

Recommended sampling

Ornith's model card recommends, for general / coding use:

{ "temperature": 0.6, "top_p": 0.95, "top_k": 20 }

(temperature: 1.0 is the benchmark-reproduction setting and is what the baked generation_config.json defaults to — 0.6 is calmer and better for agent/coding work.) Set these in your client, or put a tiny proxy in front (see the optional logproxy.py, which also fills these defaults in and logs every request's question, answer and speed).

Thinking (reasoning) control

Thinking is on by default. Control it per request:

{ "reasoning_effort": "none" }   // OpenAI-style, cleanest (thinking OFF)

Server-side, --default-thinking-budget N caps thinking to N tokens (we use 8192).

---

Using it from Goose (or any OpenAI client)

Goose is an open agent CLI. Point it at NInfer with a custom OpenAI provider. Create ~/.config/goose/custom_providers/ninfer_ornith.json (Linux/macOS) or %APPDATA%\Block\goose\config\custom_providers\ninfer_ornith.json (Windows):

{
  "name": "ninfer_ornith",
  "engine": "openai",
  "display_name": "Ornith-1.5-35B uncensored (NInfer)",
  "api_key_env": "NINFER_API_KEY",
  "base_url": "http://YOUR-SERVER:8188",
  "base_path": "v1/chat/completions",
  "models": [
    { "name": "ornith-35b", "context_limit": 262000,
      "input_token_cost": 0, "output_token_cost": 0,
      "supports_cache_control": false, "reasoning": true }
  ],
  "supports_streaming": true,
  "requires_auth": true,
  "preserves_thinking": true
}

Then:

export NINFER_API_KEY=your-key        # same string you passed to --api-key (any string if none)
goose configure                       # provider: ninfer_ornith, model: ornith-35b
goose session

Notes:

• model.name must match what the server reports (ornith-35b, or whatever you set via --model-id).
• "reasoning": true makes Goose display the model's thinking. To turn thinking OFF, send {"reasoning_effort":"none"} per request.
• Same shape works for OpenWebUI, Chatbox, LibreChat, LM Studio's OpenAI import, etc.

---

Gotchas we hit (so you don't)

• chat template hash must equal the official Qwen/Qwen3.6-35B-A3B one, or convert is rejected — swap it in (step 4a). The tokenizer itself is already identical.
• MTP experts unfused → the converter complains missing experts.gate_up_proj/down_proj, extra experts.<n>.*. Fuse them first (step 4b).
• Feed BF16, not a pre-quantized NVFP4/ModelOpt checkpoint — the 35B-A3B path quantizes internally and does not ingest external quantized weights.
• Prefer --spec mtp on a heavily retrained base; external DFlash accept is lower.
• Restarting the server rapidly can leave a zombie (process up, 0 MiB on GPU). Hard-kill with pkill -9 -x ninfer-serve, wait for GPU to read 0, then start once.

---

🇹🇷 Türkçe rehber (yeni başlayanlar için, adım adım)

Bu ne işe yarıyor?

Tek bir RTX 5090 (32 GB) ekran kartında, sansürsüz 35B-A3B (MoE) bir modeli, modelin tam 262.000 tokenlık bağlamıyla çalıştırıyoruz. "Bağlam" (context) = modelin aynı anda okuyabildiği metin miktarı; 262K token kabaca birkaç yüz sayfa demek. "A3B" = model 35 milyar parametreli ama her token için sadece ~3 milyarı çalışıyor (MoE = uzman karışımı), bu yüzden hem büyük hem hızlı. 4-bit KV cache belleği yarıya indirir, MTP spekülatif çözümleme hızı 2-3 katına çıkarır.

Kullandığımız sansürsüz taban: huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated. Hazır çalıştırılabilir .ninfer dosyası: pcmaker/Huihui-Ornith-1.5-35B-A3B-Uncensored-NInfer.

⚠️ Sadece yetkili kullanım. Sansürsüz (abliterated) modeller, modelin yerleşik "reddetme" davranışını kaldırır. Bunu yalnızca kendi uygulamalarının güvenlik testi, güvenlik araştırması ve çevrimdışı deney için kullan. Ürettiğin içerikten sen sorumlusun.

Neden bu model 27B'den farklı?

Bu model MoE (256 uzman, token başına 8 aktif) + hibrit Gated-DeltaNet mimarisi (qwen3_5_moe, Qwen3.6-35B-A3B ailesi). İki pratik fark:

1. Ayrı kuantizasyon adımı YOK. NInfer'in 35B-A3B dönüştürücüsü BF16 + DFlash alır, kuantizasyonu kendisi yapar (~22.8 GB). Hazır NVFP4 dosyası çalışmaz.
2. MTP, DFlash'ten iyi. Model kendi eğittiği MTP başlığını taşır → --spec mtp kendisiyle hizalı (kabul ~%76-83). Dışarıdan DFlash, resmi checkpoint'e göre eğitildiği için bu modelde daha düşük kalır. MTP kullan.

Adım adım (Ubuntu 24.04 + RTX 5090 + CUDA 13.3)

0) Gerekli paketler:

sudo apt-get update
sudo apt-get install -y cmake ninja-build gcc g++ pkg-config \
  libavformat-dev libavcodec-dev libavutil-dev libswscale-dev libavdevice-dev libcurl4-openssl-dev
pip install -U "huggingface_hub[cli]" transformers safetensors torch

1) NInfer'i derle (~30 dk):

git clone --depth 1 https://github.com/Neroued/ninfer.git && cd ninfer
export CUDA_HOME=/usr/local/cuda-13.3 PATH="/usr/local/cuda-13.3/bin:$PATH"
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j 6
export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:$LD_LIBRARY_PATH

2) En kolay yol — hazır dosyayı indir:

hf download pcmaker/Huihui-Ornith-1.5-35B-A3B-Uncensored-NInfer --local-dir ./art

Sonra 6. adıma (servis) geç. Kendin üretmek istersen 3-5. adımlar:

3) İki kaynağı indir (BF16):

hf download huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated --local-dir src/base-bf16   # ~72 GB
hf download z-lab/Qwen3.6-35B-A3B-DFlash                     --local-dir src/dflash

4) Dönüştürücünün takıldığı iki detayı düzelt:

# (a) chat template'i resmi Qwen3.6-35B-A3B ile değiştir (diğer 5 frontend dosyası zaten aynı)
hf download Qwen/Qwen3.6-35B-A3B chat_template.jinja --local-dir /tmp/qwtmpl
cp /tmp/qwtmpl/chat_template.jinja src/base-bf16/chat_template.jinja
# (b) MTP uzmanlarını fuse et (tek shard'ı yeniden yazar, GPU gerekmez)
python fuse_mtp_experts.py src/base-bf16

5) .ninfer dosyasına dönüştür (NInfer kendi kuantize eder, ~1 dk):

cd ninfer
python3 -m tools.convert.qwen3_6_35b_a3b.convert \
  --model ../src/base-bf16 --dflash-model ../src/dflash \
  --out ../models/ornith_35b.ninfer --device cuda

6) Servis et (OpenAI uyumlu sunucu, 8188 portu):

export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:$LD_LIBRARY_PATH
./ninfer/build/apps/ninfer-serve models/ornith_35b.ninfer \
  --host 0.0.0.0 --port 8188 --model-id ornith-35b \
  --max-context 262144 --kv-dtype k8v4 --spec mtp --draft-tokens 3 --default-thinking-budget 8192

7) Dene:

curl http://localhost:8188/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"ornith-35b","messages":[{"role":"user","content":"Merhaba, kimsin?"}],
       "max_tokens":100,"temperature":0.6,"top_p":0.95,"top_k":20,"reasoning_effort":"none"}'

Önerilen sampling (sıcaklık vb.)

Ornith model kartı genel/kod kullanımı için: temperature=0.6, top_p=0.95, top_k=20. (temperature=1.0 benchmark ayarıdır ve baked generation_config.json varsayılanıdır; günlük kullanımda 0.6 daha odaklı.) Bunları istemcide ver ya da öndeki logproxy.py ile otomatik doldur (o ayrıca her isteğin sorusunu, cevabını ve hızını loglar).

Goose ile kullanımı (Türkçe)

Yukarıdaki İngilizce bölümdeki JSON'u %APPDATA%\Block\goose\config\custom_providers\ninfer_ornith.json (Windows) veya ~/.config/goose/custom_providers/ (Linux/macOS) altına koy, base_url'ü kendi sunucun yap. "reasoning": true düşünmeyi Goose'da gösterir. Düşünmeyi kapatmak için isteklere "reasoning_effort":"none" ekle. Sonra goose configure (sağlayıcı: ninfer_ornith, model: ornith-35b) ve goose session.

Sık karşılaşılan sorunlar

• chat template hash uyuşmazlığı: resmi Qwen3.6-35B-A3B template'ini kopyala (4a).
• MTP uzmanları unfused: fuse_mtp_experts.py ile fuse et (4b).
• BF16 ver, hazır NVFP4 verme — bu yol içeride kendi kuantize eder.
• Sunucu açık ama GPU 0 MiB: pkill -9 -x ninfer-serve, GPU 0 olunca tekrar başlat.

---

Credits

Built by the community, for the community. Thanks to the Ornith team, huihui-ai, the Qwen team, z-lab, and the NInfer author (Neroued). This guide documents an integration; all model weights and tools belong to their respective authors and licenses.