Qwen3.8-27B-Uncensored-NInfer-RTX5090-Guide

Qwen3.8‑27B Uncensored @ 262K context on a single RTX 5090 — with NInfer

A complete, reproducible guide to run an uncensored Qwen3.8‑27B at the model's full native 262,144‑token context on one consumer RTX 5090 (32 GB), using 4‑bit (NVFP4/K8V4) KV cache, host‑RAM KV offload and DFlash2 speculative decoding — reaching ~120–180 tok/s on a single GPU.

The uncensored base we used is huihui‑ai/Huihui‑Qwen3.8‑27B‑abliterated. A ready‑to‑run .ninfer artifact of it is here: pcmaker/Huihui‑Qwen3.8‑27B‑Uncensored‑NInfer.

This document exists because getting all of these to work at the same time on a single 32 GB card is not obvious. The mainstream engine (SGLang) could not do it for this specific architecture in late 2026; a purpose‑built engine (NInfer) could. Everything here is open source. Nothing below contains any private key, password or host.

Intended use. Uncensored / abliterated weights remove the model's built‑in refusals. This guide is for authorized work only — e.g. red‑teaming and security testing of your own applications, safety research, and offline experimentation. You are responsible for what you generate. Do not use it to harm people.

---

TL;DR — what we achieved

Quality is unchanged vs. the base uncensored model — speculative decoding is verified by the main model, so wrong drafts are discarded.

---

Hardware & OS that this ran on

• GPU: NVIDIA GeForce RTX 5090, 32 GB, Blackwell, compute capability sm_120.
• OS: Ubuntu 24.04 LTS (64‑bit).
• CUDA Toolkit: 13.3 (nvcc 13.3). A Blackwell card + CUDA 13.x is required.
• CPU/RAM: any modern desktop; we used a 12‑core Ryzen with 128 GB DDR5.
• Disk: ~200 GB free during the build (BF16 base ~55 GB + quantized copies + ~24 GB artifact).

---

Why NInfer (and not vLLM / SGLang / TensorRT‑LLM)

Qwen3.8‑27B is a hybrid Gated DeltaNet (GDN) + Gated Attention model: of its 64 layers, only 16 are full‑attention (they keep a KV cache); the other 48 are linear‑ attention (Gated DeltaNet, no KV cache). This is great for long context, but the tooling in late 2026 was immature for it:

• SGLang 0.5.18 loads the model, but its NVFP4 KV‑cache path hit KeyError: 'float4_e2m1fn_x2' on the only backend (triton) that supports both GDN and 4‑bit KV on Blackwell. So on SGLang we were stuck at fp8 KV and ~131k context.
• TensorRT‑LLM can reject hybrid Mamba/GDN NVFP4 models at startup (mamba state slots mis‑sized) — same wall, plus heavy engine builds.
• NInfer (github.com/Neroued/ninfer) is a from‑scratch C++20/CUDA engine purpose‑built for a handful of Qwen checkpoints on a single RTX 5090. It natively supports GDN, NVFP4 weights, 4‑bit KV, host‑RAM KV offload and DFlash2. This is the piece that made everything fit.

---

Open‑source components used

---

Step‑by‑step

0. Prerequisites (Ubuntu 24.04)

sudo apt-get update
sudo apt-get install -y cmake ninja-build gcc g++ pkg-config \
  libavformat-dev libavcodec-dev libavutil-dev libswscale-dev \
  libavdevice-dev libcurl4-openssl-dev
nvcc --version   # -> release 13.3
pip install -U llmcompressor transformers accelerate datasets torch

1. Build NInfer

git clone --depth 1 https://github.com/Neroued/ninfer.git && cd ninfer
export CUDA_HOME=/usr/local/cuda-13.3 PATH="/usr/local/cuda-13.3/bin:$PATH"
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j 6            # ~30 min, 347 targets
export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:$LD_LIBRARY_PATH

2. Download the three conversion sources

hf download huihui-ai/Huihui-Qwen3.8-27B-abliterated --local-dir src/base-bf16
hf download unsloth/Qwen3.8-27B-NVFP4                --local-dir src/ref-nvfp4
hf download z-lab/Qwen3.8-27B-DFlash2                --local-dir src/dflash2

Critical — the base's tokenizer must be byte‑identical to Qwen/Qwen3.8-27B. Many uncensored fine‑tunes ship a modified tokenizer and are rejected (tokenizer.json hash mismatch). huihui-ai and orcarouter are fine; heretic‑ara, PristinelyUncensored and JonathanColetti are not.

3. Quantize the uncensored weights to the exact NVFP4 layout

See quantize_uncensored_nvfp4.py and make_calibration.py.

python make_calibration.py
python quantize_uncensored_nvfp4.py huihui-ai/Huihui-Qwen3.8-27B-abliterated src/unc-nvfp4

The produced config.json must report format: mixed-precision, a group_0 FP8 and a group_1 NVFP4.

4. Convert to a .ninfer artifact

cd ninfer
python3 -m tools.convert.qwen3_8_27b.convert_nvfp4 \
  --model ../src/base-bf16 --quantized-model ../src/unc-nvfp4 \
  --dflash2-model ../src/dflash2 --out ../models/qwen38_uncensored.ninfer --device cuda

Expect preflight complete: 1190 objects, 233 FP8 and 168 NVFP4 source matrices, 81 DFlash2 source tensors; ~1 min, ~24 GB artifact.

5. Serve it

See serve_ninfer.sh:

export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:$LD_LIBRARY_PATH
./ninfer/build/apps/ninfer-serve models/qwen38_uncensored.ninfer \
  --host 0.0.0.0 --port 8188 --max-context 262144 \
  --kv-dtype k8v4 --spec dflash2 --draft-tokens 3 --default-thinking-budget 600

OpenAI‑ and Anthropic‑compatible endpoints on port 8188. Useful flags: --api-key, --model-id <name>, --host-kv-mib N, --kv-dtype nvfp4|k8v4|fp8|int8|bf16.

---

Using it from Goose (or any OpenAI client)

Goose is an open agent CLI. Point it at NInfer with a custom OpenAI provider. Create ~/.config/goose/custom_providers/ninfer_qwen.json (Linux/macOS) or %APPDATA%\Block\goose\config\custom_providers\ninfer_qwen.json (Windows):

{
  "name": "ninfer_qwen",
  "engine": "openai",
  "display_name": "Qwen3.8-27B uncensored (NInfer)",
  "api_key_env": "NINFER_API_KEY",
  "base_url": "http://YOUR-SERVER:8188",
  "base_path": "v1/chat/completions",
  "models": [
    { "name": "qwen3.8-27b", "context_limit": 262000,
      "input_token_cost": 0, "output_token_cost": 0,
      "supports_cache_control": false, "reasoning": false }
  ],
  "supports_streaming": true,
  "requires_auth": true
}

Then set the key and select the provider:

export NINFER_API_KEY=your-key        # the same key you passed to --api-key (any string if none)
goose configure                       # pick provider: ninfer_qwen, model: qwen3.8-27b
goose session

Notes:

• The model.name must match what the server reports. It is the artifact identity (qwen3.8-27b) unless you started NInfer with --model-id <something-else>.
• base_url is your NInfer host; base_path is v1/chat/completions. Same shape works for OpenWebUI, Chatbox, LibreChat, LM Studio's OpenAI import, etc.
• Thinking: NInfer's thinking is on by default. Keep --default-thinking-budget 600 on the server so it never runs away, and/or send {"reasoning_effort":"none"} per request. Some clients let you add default request fields; if yours can't, the budget flag alone is a fine compromise.

---

Thinking (reasoning) control

Control it per request (no proxy needed):

{ "reasoning_effort": "none" }                      // OpenAI-style, cleanest
{ "enable_thinking": false }
{ "chat_template_kwargs": { "enable_thinking": false } }

Server‑side, --default-thinking-budget N caps thinking to N tokens.

---

Gotchas we hit (so you don't)

• Tokenizer hash must match official Qwen3.8 or convert is rejected.
• The --quantized-model must be the mixed FP8+NVFP4 layout — a single‑group NVFP4 is not accepted by the converter.
• --no-thinking as a server flag was unstable; use --default-thinking-budget + per‑request reasoning_effort.
• Restarting the server rapidly can leave a zombie (process up, 0 MiB on GPU). Hard‑kill, wait for GPU to read 0, remove the old log, then start once and wait for engine ready.
• A tensor descriptor does not match target contract error means the artifact was built with a different NInfer version — build your own with the same tree you serve on.

---

🇹🇷 Türkçe rehber (yeni başlayanlar için, adım adım)

Bu ne işe yarıyor?

Tek bir RTX 5090 (32 GB) ekran kartında, sansürsüz Qwen3.8‑27B modelini modelin tam 262.000 tokenlık bağlamıyla çalıştırıyoruz. "Bağlam" (context) = modelin aynı anda okuyabildiği metin miktarı; 262K token kabaca birkaç yüz sayfa demek. Ayrıca 4‑bit KV cache (belleği yarıya indirir), KV'yi RAM'e taşıma (VRAM'i rahatlatır) ve DFlash2 spekülatif çözümleme (2‑3 kat hız) kullanıyoruz. Sonuç: ~120‑180 token/sn.

Kullandığımız sansürsüz taban model: huihui‑ai/Huihui‑Qwen3.8‑27B‑abliterated. Hazır çalıştırılabilir .ninfer dosyası: pcmaker/Huihui‑Qwen3.8‑27B‑Uncensored‑NInfer.

⚠️ Sadece yetkili kullanım. Sansürsüz (abliterated) modeller, modelin yerleşik "reddetme" davranışını kaldırır. Bunu yalnızca kendi uygulamalarının güvenlik testi, güvenlik araştırması ve çevrimdışı deney için kullan. Ürettiğin içerikten sen sorumlusun; kimseye zarar verme.

Neden NInfer? (kısa hikaye)

Qwen3.8‑27B "hibrit" bir mimari: 64 katmanın yalnızca 16'sı klasik dikkat (KV cache tutar), 48'i lineer dikkat (Gated DeltaNet, KV tutmaz). Bu, uzun bağlam için harika ama 2026 sonunda yazılım desteği olgun değildi. SGLang'de 4‑bit KV yolu Blackwell'de bir hataya (float4_e2m1fn_x2) takılıp 131K'da kalıyordu. NInfer bu mimari için sıfırdan yazılmış olduğu için 262K + 4‑bit KV + DFlash2'yi tek kartta çözdü.

Adım adım (Ubuntu 24.04 + RTX 5090 + CUDA 13.3)

0) Gerekli paketler:

sudo apt-get update
sudo apt-get install -y cmake ninja-build gcc g++ pkg-config \
  libavformat-dev libavcodec-dev libavutil-dev libswscale-dev \
  libavdevice-dev libcurl4-openssl-dev
pip install -U llmcompressor transformers accelerate datasets torch

1) NInfer'i derle (yaklaşık 30 dakika):

git clone --depth 1 https://github.com/Neroued/ninfer.git && cd ninfer
export CUDA_HOME=/usr/local/cuda-13.3 PATH="/usr/local/cuda-13.3/bin:$PATH"
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j 6
export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:$LD_LIBRARY_PATH

2) En kolay yol — hazır dosyayı indir (quantize/convert ile uğraşmadan):

hf download pcmaker/Huihui-Qwen3.8-27B-Uncensored-NInfer --local-dir ./art

Sonra 5. adıma geç (servis et). Kendin üretmek istersen 3‑4. adımlar:

3) Kaynakları indir ve kendin kuantize et (ileri seviye):

hf download huihui-ai/Huihui-Qwen3.8-27B-abliterated --local-dir src/base-bf16
hf download unsloth/Qwen3.8-27B-NVFP4                --local-dir src/ref-nvfp4
hf download z-lab/Qwen3.8-27B-DFlash2                --local-dir src/dflash2
python make_calibration.py
python quantize_uncensored_nvfp4.py huihui-ai/Huihui-Qwen3.8-27B-abliterated src/unc-nvfp4

⚠️ Önemli: seçtiğin sansürsüz tabanın tokenizer'ı resmi Qwen3.8 ile birebir aynı olmalı. huihui‑ai ve orcarouter uygundur; heretic/Zynerji gibi tokenizer'ı değiştirenler dönüştürmede reddedilir.

4) .ninfer dosyasına dönüştür:

cd ninfer
python3 -m tools.convert.qwen3_8_27b.convert_nvfp4 \
  --model ../src/base-bf16 --quantized-model ../src/unc-nvfp4 \
  --dflash2-model ../src/dflash2 --out ../models/qwen38_uncensored.ninfer --device cuda

5) Servis et (OpenAI uyumlu sunucu, 8188 portu):

export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:$LD_LIBRARY_PATH
./ninfer/build/apps/ninfer-serve ./art/huihui_uncensored_nvfp4.ninfer \
  --host 0.0.0.0 --port 8188 --max-context 262144 \
  --kv-dtype k8v4 --spec dflash2 --draft-tokens 3 --default-thinking-budget 600

6) Dene (başka bir terminalden):

curl http://localhost:8188/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"Merhaba, kimsin?"}],"max_tokens":100,"reasoning_effort":"none"}'

Goose ile kullanımı (Türkçe)

Goose açık kaynak bir yapay zeka ajan aracı. NInfer'e bağlamak için yukarıdaki İngilizce bölümdeki JSON'u ~/.config/goose/custom_providers/ninfer_qwen.json (Windows'ta %APPDATA%\Block\goose\config\custom_providers\) altına koy, base_url'ü kendi sunucunun adresi yap. Sonra:

export NINFER_API_KEY=anahtarin          # sunucuda --api-key ile aynı (yoksa herhangi bir şey)
goose configure                          # sağlayıcı: ninfer_qwen, model: qwen3.8-27b
goose session

Düşünme (thinking) modunu kapatmak için isteklere "reasoning_effort":"none" ekle; sunucuda --default-thinking-budget 600 zaten aşırı düşünmeyi engeller.

Sık karşılaşılan sorunlar

• tokenizer hash uyuşmazlığı: sansürsüz taban resmi Qwen3.8 tokenizer'ını değiştirmiş demektir; huihui‑ai veya orcarouter kullan.
• Sunucu açık ama GPU 0 MiB: hızlı yeniden başlatma zombi süreç bırakır. Süreci sert öldür, GPU 0 olana kadar bekle, eski logu sil, tek seferde başlat ve engine ready yaz.
• tensor descriptor does not match: dosya farklı bir NInfer sürümüyle üretilmiş; servis ettiğin ağaçla aynı sürümde kendin üret.

---

Credits

Built by the community, for the community. Thanks to the Qwen team, huihui‑ai, orcarouter, unsloth, z‑lab, the NInfer author (Neroued), and the llm‑compressor project. This guide documents an integration; all model weights and tools belong to their respective authors and licenses.