Qwen3.8-27B-Huihui-Abliterated-NInfer-NVFP4

Huihui Qwen3.8-27B Abliterated — NInfer Mixed FP8/NVFP4

A complete NInfer artifact derived from huihui-ai/Huihui-Qwen3.8-27B-abliterated revision d42ca8978c5a66e92c3446d46e8adfe03ef692ff.

The artifact combines NInfer's fixed FP8 + NVFP4 Qwen3.8 allocation with same-source Huihui BF16 norms, embeddings, vision tower, and MTP tensors. No official Qwen or unrelated behavioral checkpoint supplies model weights.

Choose a compatible artifact

Both formats are available in this repository. Download only the file matching your runtime.

The v3 file preserves all 1,118 original tensor encodings and the entire original payload byte-for-byte. It appends the maintained Qwen3.8 chat template and adds the v3 directory and bindings. This is a container/template update, not a new quantization. It does not add an indexed proposal head or DFlash2 weights; omit --lm-head-draft and --spec dflash2.

The original file keeps its filename, bytes and SHA256. Existing pinned downloads continue to work. The v3 file requires a v3 runtime; the original v2 file requires its legacy runtime. Do not download the whole repository unless you want both files. Native Hugging Face Xet uploads use content-defined chunk deduplication, which can reuse the preserved payload across these files; logical file sizes are not a measurement of physical backend storage or billable usage.

v3 download and run

hf download lyf/Qwen3.8-27B-Huihui-Abliterated-NInfer-NVFP4 \
  qwen3_8_27b_nvfp4.v3.ninfer --local-dir ./huihui-ninfer

git clone https://github.com/Neroued/ninfer.git
cd ninfer
git checkout 4de9e42599b3f1fb893cac64a063e6f7d2581515
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
./build/apps/ninfer-serve ../huihui-ninfer/qwen3_8_27b_nvfp4.v3.ninfer \
  --host 127.0.0.1 --port 8000 \
  --model-id qwen38-huihui-ninfer-nvfp4 \
  --max-context 212992 --kv-capacity auto --max-concurrency 4 \
  --prefill-chunk 1024 --kv-dtype fp8 \
  --spec mtp --draft-tokens 3 --vision \
  --default-max-tokens 4096 --preserve-thinking

The runtime is deliberately pinned before the context-refactor regression reported in NInfer #414. This does not establish that the revision is free of all bugs. Build requirements are in the pinned upstream README; this release was built with CUDA 13.1 on Linux and tested on one RTX 5090 at a 450 W power limit. Vision is optional; omit --vision for text-only use. KV capacity is shared across concurrent requests; four lanes do not imply four simultaneous 212,992-token contexts.

See BUILD_MANIFEST.v3.json, VALIDATION_REPORT.v3.json, PAYLOAD_VERIFICATION.v3.json, and SHA256SUMS.v3 for provenance and verification. Legacy measurements below describe the original August release, not this runtime.

Xet upload reuse

The v3 file is 21.49 GB logically. The native Xet upload progress reported about 584 MB of new data; completed upload calls recorded 574,181,890 bytes of new xorb objects plus 1,060,560 bytes of shard metadata. Dedup queries found existing content. These counters demonstrate reuse, not billable physical storage or total network traffic. See XET_UPLOAD_REPORT.v3.json.

v3 validation (2026-10-26)

The profile above passed text, vision, separate thinking output, SSE, JSON Schema, required tool calling, OpenAI Responses and Anthropic Messages checks. Four simultaneous requests with the same 1,555-token prompt each completed 512 output tokens; the server remained healthy, including after prefix-checkpoint reuse.

A repeated-text retrieval fixture actually submitted 204,042 Chat prompt tokens, generated the correct marker from the beginning, and completed without OOM: prefill 47.75 seconds, 4,273 tok/s. This is one synthetic retrieval fixture, not broad long-context quality validation. Single-seed short smoke decode rates were 145 tok/s (Chinese, 256 output tokens), 195 tok/s (code, 256 output tokens), and 176 tok/s (vision, 165 output tokens). The 256-token text/code outputs were truncated. These are smoke measurements, not a legacy-runtime A/B or a general model-quality benchmark.

Provenance clarification

The legacy converter's qwen3_8_27b_nvfp4.ninfer.conversion.json contains hard-coded official Qwen/unsloth repository and revision labels despite using the local Huihui source paths. The actual derivative source is Huihui revision d42ca8978c5a66e92c3446d46e8adfe03ef692ff, as recorded in BUILD_MANIFEST.json and the original release process. The v3 release is derived directly from the published legacy artifact, with exact payload verification; no official target weights are introduced during migration. The historical conversion log is retained unchanged.

Legacy v2: original release

hf download lyf/Qwen3.8-27B-Huihui-Abliterated-NInfer-NVFP4 qwen3_8_27b_nvfp4.ninfer   --local-dir ./huihui-ninfer

git clone https://github.com/Neroued/ninfer.git
cd ninfer
git checkout a99407c63fc5bbd25d9fb597cbb8ab352bdb01ef
cmake -S . -B build -G Ninja   -DCMAKE_BUILD_TYPE=Release   -DCMAKE_CUDA_ARCHITECTURES=120a
cmake --build build -j"$(nproc)"

./build/apps/ninfer ../huihui-ninfer/qwen3_8_27b_nvfp4.ninfer   --prompt "Explain FP8 and NVFP4 briefly."   --max-new 128 --max-context 8192 --kv-dtype int8 --no-thinking

OpenAI-compatible server

./build/apps/ninfer-serve ../huihui-ninfer/qwen3_8_27b_nvfp4.ninfer   --host 127.0.0.1 --port 8000   --model-id qwen38-huihui-ninfer-nvfp4   --max-context 204800 --kv-capacity 204800   --max-concurrency 1 --prefill-chunk 4096   --kv-dtype int8 --spec mtp --draft-tokens 3   --default-max-tokens 16384 --preserve-thinking

This 204.8K server profile was startup- and API-smoke-tested on one RTX 5090. Long-context quality and maximum sustained prompt length remain workload-dependent.

Mixed allocation

NInfer source preflight passed with:

FP8 source matrices: 233
NVFP4 source matrices: 168
source fields: 1587
F8_E4M3: 401
BF16: 682
U8: 168
F32: 336
NINF_PREFLIGHT_OK

The FP8 matrices are deterministic row-scaled E4M3 exports from Huihui BF16. The NVFP4 matrices come from the separately calibrated Huihui ModelOpt NVFP4 checkpoint (CNN/DailyMail 3.0.0, 20 × 8192 tokens).

Artifact inventory

NInfer commit: a99407c63fc5bbd25d9fb597cbb8ab352bdb01ef
model_id: qwen3.8-27b
weights_id: nvfp4
objects: 1124 (1118 tensors, 6 resources)
artifact bytes: 21,492,695,040
SHA256: f21f308d3b23ccd627071cd015e413db08deee4356643900518e2b251750fdc2

Key formats:

BF16: 534
FP32: 208
FP8 row-scaled: 146
NVFP4: 112

Runtime validation

Hardware: RTX 5090 / SM120, driver 610.43.02, CUDA 13.1, 450 W cap.

Text smoke test:

prompt tokens: 27
generated tokens: 114
prefill: 792.69 tok/s
decode: 74.83 tok/s
overall: 73.83 tok/s
GPU weights: 18.98 GiB

Real-image test passed. For the test image, the model correctly identified a red square in the upper-left and a blue circle in the lower-right. Vision decode measured 74.82 tok/s with 19.25 GiB of GPU weights loaded.

A benign ablation/refusal-vector research prompt also received a direct technical answer. This is a smoke test of behavioral continuity, not a comprehensive behavioral benchmark.

Included evidence

• qwen3_8_27b_nvfp4.ninfer.conversion.json
• ninfer-nvfp4-preflight.log
• artifact-inspect.log
• text, vision, and behavior outputs/logs
• BUILD_MANIFEST.json, VALIDATION_REPORT.json, SHA256SUMS

Intended use

This is an abliterated behavioral derivative intended for model research and local inference. Users are responsible for downstream use and applicable policies.