Qwen3.8-27B-nvfp4full-dflash2-NInfer

📦 v3 container available. For NInfer engines at upstream v3 (master >= f76e19c0, 2026-09-13): the bytes-preserved v3 build of this artifact is at kaushikvira/Qwen3.8-27B-nvfp4full-dflash2-NInfer-v3 (qwen3_8_27b_nvfp4full-dflash2.v3.ninfer, sha256 74d2fc17…d5fdfc) and needs no engine patch — the C++ nvfp4full profile registration is only required by this v2 container. Full lineage chain: v3-upgrade.json in the v3 repo.

Qwen3.8-27B nvfp4full + DFlash2 for NInfer

This repository contains the nvfp4full weight profile of Qwen3.8-27B in the native NInfer .ninfer artifact format, with the z-lab DFlash2 speculative-decoding drafter module added in the upstream W8G32/BF16 format.

It is the qwen3.8-27b / nvfp4full identity (same as cometkim/Qwen3.8-27B-nvfp4full-NInfer) with the registered DFlash2 module appended — the base tensors are byte-for-byte identical to cometkim v1, and the MTP module is retained (unused and validate-only under --spec dflash2).

⚠️ Format note. Unlike gpillon/Qwen3.8-27B-nvfp4full-dflash2-NInfer (NVFP4 weight-only drafter), the DFlash2 module here uses the upstream W8G32_F16S / BF16 encoding, which is the format our fork (upstream master + 1 commit) binds. The two artifacts are not interchangeable.

Artifact

Verify a downloaded file with:

printf '%s  %s\n' \
  '373dab08733f96555d964e02c6cebe40b23f753c0b2c4592fe130d641f937de1' \
  'qwen3_8_27b_nvfp4full-dflash2.ninfer' | sha256sum --check

Credits

Lineage / provenance

The base tensors are copied from cometkim v1 byte-for-byte (no re-encoding); only the 66 DFlash2 module objects are produced by the graft tool from the z-lab checkpoint. Verified: source identity qwen3.8-27b/nvfp4full, 1,259 source objects, no pre-existing dflash2/* objects, output = 1,325.

Requirements

• NInfer main (= upstream 487f8977 + commit 2eb59dbc), or upstream master at/after DFlash2 binding (385b30ce) plus the qwen3.8-27b/nvfp4full weights registration — upstream Neroued/ninfer alone does not register the nvfp4full profile yet;
• 64-bit Linux;
• NVIDIA GeForce RTX 5090 (sm_120a);
• CUDA Toolkit 13.1 or newer.

Download and run a CLI example

hf download kaushikvira/Qwen3.8-27B-nvfp4full-dflash2-NInfer \
  qwen3_8_27b_nvfp4full-dflash2.ninfer \
  --local-dir models

./build/apps/ninfer models/qwen3_8_27b_nvfp4full-dflash2.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 32768 \
  --max-new 8192 \
  --kv-dtype k8v4 \
  --spec dflash2 --draft-tokens 7

Start a local server

./build/apps/ninfer-serve models/qwen3_8_27b_nvfp4full-dflash2.ninfer \
  --host 127.0.0.1 \
  --port 8080 \
  --max-context 262144 \
  --kv-capacity auto \
  --max-concurrency 4 \
  --kv-dtype k8v4 \
  --spec dflash2 --draft-tokens 7 \
  --preserve-thinking

--spec dflash2 --draft-tokens 7 selects the DFlash2 drafter (K=7, block width 8; K=1..15 supported for the z-lab module, 7 recommended). A full sanitized serving profile (same flags, no secrets) is at examples/ninfer-nvfp4full-grafted-dflash2.cfg.example in the engine fork. Leave --lm-head-draft off (the full-vocabulary proposal head is the default and DFlash2 candidates span the whole vocabulary). --spec mtp still works — the MTP module is retained.

Supported use

• text generation in thinking and non-thinking modes;
• image, multi-image, video, and mixed multimodal messages (--vision);
• DFlash2 speculative decoding, draft windows 1–15 (recommended 7);
• MTP speculative decoding (--spec mtp, windows 1–5);
• BF16, INT8, FP8, NVFP4, and K8V4 KV cache (--kv-dtype);
• CUDA Graph decode and compatible-prefix reuse;
• startup-bounded small-scale concurrent serving (1–8 active requests);
• the NInfer CLI and OpenAI/Anthropic-compatible HTTP serving.

Performance (measured on our rig, 2026-09-06)

One NVIDIA GeForce RTX 5090 (32 GB, 450 W cap), CUDA 13.1, DFlash2 K=7, --kv-dtype k8v4, --max-context 262144, --max-concurrency 4, vision on. Probes: bench/tests/perf.py, bench/tests/conc_perf.py, bench/tests/needle.py (same probes used for the engine A/B).

Same rig, same probes — the reference profile (cometkim nvfp4full + MTP4 + k8v4, our previous base) decodes at 157.9 tok/s with needle 12/12; this artifact is +11% decode at equal quality on the same KV codec.

These are our own numbers (single sample per probe, deterministic where possible) — not vendor benchmarks; reproduction commands in the repository.

Limits

• Accepted only by a NInfer build that registers qwen3.8-27b/nvfp4full (our fork, or upstream after the registration lands);
• one RTX 5090 / one CUDA device, startup-fixed capacity 1–8 requests;
• no multi-GPU, offload, preemptive batching, or distributed serving;
• context allocation is subject to GPU memory and the KV-cache type — on a 32 GB card with k8v4 the auto KV pool lands at ~284k tokens, still ≥ 1 full 262,144-token context; host-RAM continuation reuse is a server flag, not this artifact;
• NInfer does not execute generated tool calls.

Superseded: the v2 container is retained for history only; the v3 engine rejects v2 artifacts by design. Use the v3 artifact.

kaushikvira Qwen3.8-27B artifact family (single-RTX-5090 serving)

All artifacts are NInfer containers of Qwen3.8-27B derivatives, served by the same engine on the same box. Per-card A/B tables were measured pairwise in the same session; cross-card rows are from different sessions — treat small deltas as indicative.

* values for the nvfp4full-dflash2.v3 row come from its same-session A/B against swift-abliterated (66.3-vs-65.0 etc. are paired measurements, not independent runs).

llama-benchy (single RTX 5090, engine-native API)

Measured on the v3 twin of this exact artifact (Qwen3.8-27B-nvfp4full-dflash2-NInfer-v3 — byte-identical weights, container-format upgrade only; the v2 container itself is not servable by the current v3 engine).

See the v3 card for the full measurement-conditions block (450 W cap, SM clock pin, telemetry).

---

Acceptable Use Policy & Legal Notice

By downloading, possessing, or using this artifact you agree to the terms below.

Permitted use

Research, personal/local experimentation, red-teaming, safety research, and evaluating alignment and quantization techniques — subject to all applicable laws and regulations.

Prohibited use

You must not use this model, alone or in any pipeline, to:

• carry out, plan, or facilitate any activity that is illegal in your jurisdiction;
• produce content that harms, endangers, defrauds, harasses, or exploits others — including but not limited to weapons, malware, exploitation of minors, targeted harassment, or disinformation presented as fact;
• provide medical, legal, or financial advice presented as professionally qualified;
• violate the rights of any person or entity, including intellectual property and privacy rights.

Your responsibility

Model outputs are a capability, not a judgment — never guaranteed trustworthy, factual, or legal. You are solely and fully responsible for every prompt you send, every output you generate, and every use you make of them. The publisher of this artifact (kaushikvira) does not monitor, endorse, or take any part in downstream use.

No warranty / limitation of liability

The artifact is provided "AS IS", WITHOUT WARRANTY OF ANY KIND, express or implied, including merchantability, fitness for a particular purpose, and non-infringement. To the maximum extent permitted by applicable law, the publisher shall not be liable for any claim, damages, or other liability, whether in contract, tort or otherwise, arising from, out of, or in connection with this artifact or its use. Nothing in this card limits liability where limitation is not permitted by law.

Reporting

If you become aware of misuse of this model, report it to the platform where the misuse occurs and to the relevant authorities.

License

Apache-2.0. Base repositories, quantized source, and the DFlash2 drafter are also Apache-2.0.