Downloads · 30 days
20
3% of all-time downloads
engharat2/Qwen3.8-27B-QUASAR-NVFP4-NINFER
Qwen3.8-27B-QUASAR-NVFP4-NINFER is a text generation model from engharat2. Use it when you need the model to write or continue text. It is set up for custom. The card lists the license as apache-2.0.
The QUASAR QAT NVFP4 (W4A4) variant of Qwen3.8-27B, converted into the closed .ninfer artifact format used by the NInfer C++/CUDA inference engine, and evaluated on GPQA-Diamond.
Downloads · 30 days
20
3% of all-time downloads
All-time downloads
761
Public
Repo size
18.7 GB
Likes
0
Public
Click a slice to open those files.
.ninfer18.7 GB · 100%
From the Hugging Face model README
The QUASAR QAT NVFP4 (W4A4)
variant of Qwen3.8-27B, converted into the closed .ninfer artifact format
used by the NInfer C++/CUDA inference
engine, and evaluated on GPQA-Diamond.
This is a compiled inference artifact, not a transformers-loadable checkpoint. The single file
qwen3_8_27b_nvfp4_quasar.ninfer(17.36 GiB) is loaded byninfer/ninfer-serve, not bytransformers. See serving.
Update: I would like to underline that the behavior of the model in actual usage is a bit...weird, with some allucinations happening. As I haven't tested the original QUASAR model, I don't know if the issue comes from my porting, or from QUASAR quantization, or from original Qwen 3.8 NVFP4 model.
| File | Size | What |
|---|---|---|
qwen3_8_27b_nvfp4_quasar.ninfer | 18,668,301,312 B | the artifact (1,268 objects: 1,262 tensors + 6 frontend resources) |
config.json | — | QUASAR model config + base_model lineage + NInfer serving identity |
PROVENANCE.json | — | machine-readable pins: sources, fork commit, artifact SHA-256, benchmark |
Artifact SHA-256: e75ce1706f4ddd90a4d8a432c3609f23280034f9ff7a804de1e739e948b60c89
Qwen/Qwen3.8-27B (BF16 original)
└─ QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 (QAT NVFP4 W4A4, vLLM) ← base_model
└─ engharat2/Qwen3.8-27B-QUASAR-NVFP4-NINFER (this repo, NInfer artifact)
Converted with the dual-source converter committed to
engharat/ninfer at e80b1ed5
(fork of Neroued/ninfer). Full
reproduction steps, design decisions, and per-sample benchmark data live in
the documentation repo.
| Format | Count | What |
|---|---|---|
NVFP4 (blockscale-k16-m128x4-v1) | 256 | all 400 fused/standalone text linears' packed weights |
BF16 (contiguous-le-v1) | 535 | norms, token_embedding, GDN conv / a_b, vision, MTP norms |
FP32 (contiguous-le-v1) | 352 | A_log, dt_bias, scales, draft-head ids |
FP8_E4M3FN_ROW_BF16S (row-scale-v1) | 1 | output_head (lm_head) |
Q4/Q5/Q6G64_F16S (row-split-k128-v1) | 55 / 54 / 1 | vision tower + draft head |
W8G32_F16S (row-split-k128-v1) | 7 | 5 MTP projections + 2 vision |
I32 | 1 | draft-head token ranking |
All 400 non-control text linears are bit-identical NVFP4 to the QUASAR
source (same packed E2M1 codes, FP8 group scales, and static QAT
input_global_scale values). The only deliberate precision changes vs the
source: GDN control projections (in_proj_a/in_proj_b, 48 layers) are
dequantized to BF16 at conversion (neutral-to-positive), and output_head
is row-scaled FP8 (~3% relative logit noise, the sole real compromise).
GPQA-Diamond — the formal NInfer Qwen3.8-27B NVFP4 protocol (198 samples,
0-shot, thinking on, temperature 1.0 / top_p 0.95 / top_k 20, seed 42,
concurrency 1, rule-scored). Two independent full runs on this served
artifact (MTP spec decoding on, BF16 KV @ 194,560 context):
| Correct | Score | |
|---|---|---|
| Run 1 | 179 / 198 | 0.9040 |
| Run 2 | 179 / 198 | 0.9040 |
| Pooled (n=396) | 358 / 396 | 0.9040 |
Versus the QUASAR model card's vLLM references (same n=396 pooling):
BF16 original 0.9141 (362/396), QUASAR via vLLM 0.9091 (360/396),
this NInfer artifact 0.9040 (358/396), unsloth NVFP4 0.8939 (354/396),
Inferact NVFP4 0.8763 (347/396). The 2-sample gap vs the vLLM QUASAR run is
inside the benchmark's sampling noise (binomial σ ≈ 3.7 at n=396, p≈0.91)
and tracks the FP8 lm_head.
It requires a fork of NInfer due to added QUASAR model support and due to custom chat template jinja support.
# NInfer fork at the pinned commit, built:
git clone https://github.com/engharat/ninfer && cd ninfer
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && cmake --build build --parallel
# Fetch this artifact and serve on an RTX 5090 (32 GB):
hf download engharat2/Qwen3.8-27B-QUASAR-NVFP4-NINFER --local-dir ./out
./build/apps/ninfer-serve out/qwen3_8_27b_nvfp4_quasar.ninfer \
--host 0.0.0.0 --port 8080 \
--max-context 194560 --max-concurrency 2 \
--spec mtp --draft-tokens 3 --lm-head-draft \
--pending-timeout-ms 300000
Observed: ~5–8k tok/s prefill, ~180–225 tok/s decode (MTP acceptance
~50–70% depending on generation length). --kv-dtype int8 doubles the KV
token count at the cost of 8-bit KV.
Apache-2.0. Derived from the QUASAR NVFP4 checkpoint (itself derived from
Qwen/Qwen3.8-27B); see QUASAR's model card
for the upstream licensing terms.