Downloads · 30 days
717
25% of all-time downloads
mtrskeen/qwen3.8-27b-ninfer-minq4-mtpq4-visionq4-5080
qwen3.8-27b-ninfer-minq4-mtpq4-visionq4-5080 is a image-text-to-text model from mtrskeen. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for ninfer. The card lists the license as apache-2.0.
This repository contains one NInfer .ninfer artifact for Qwen3.8-27B, converted and requantized for a single RTX 5080 16 GB (sm120) by the fork mtrskeen/ninfer-5080, which extends Neroued/ninfer and the A5000 16 GB fork
Downloads · 30 days
717
25% of all-time downloads
All-time downloads
2.9K
Public
Repo size
15.5 GB
Likes
1
Public
Click a slice to open those files.
.ninfer15.5 GB · 100%
From the Hugging Face model README
This repository contains one NInfer .ninfer artifact for
Qwen3.8-27B, converted and
requantized for a single RTX 5080 16 GB (sm_120) by the fork
mtrskeen/ninfer-5080, which extends
Neroued/ninfer and the A5000 16 GB fork
aljazceru/ninfer.
The artifact is intended only for NInfer. It is not a Transformers checkpoint, safetensors distribution, GGUF, or MLX file.
| Field | Value |
|---|---|
| Filename | qwen3_8_27b_minq4_mtpq4_visionq4.ninfer |
| Size | 15,514,935,296 bytes (14.45 GiB) |
| SHA-256 | c7eb6fdde74bfd70e168a9ccce113b08fb34ad7ce3349ea989590c4830591fb8 |
| Container version | 2 (NINFER\0\x02) |
| NInfer target key | qwen3_8_27b |
| Weights ID | groupwise-int |
| Stored objects | 1,115 tensors + 7 resources (per server load report) |
| Max context tested | 124,928 tokens (text); 32,768 (vision) |
Contents versus the registered full-precision-preserving profile:
linear_add, Q4 output head); token embedding
is stored Q6. Quality gate: simulated PPL +0.04 versus the IQ3_XXS anchor.Q4G64_F16S.Q4G64_F16S; Vision buffers
are allocated lazily.sha256sum qwen3_8_27b_minq4_mtpq4_visionq4.ninfer
# c7eb6fdde74bfd70e168a9ccce113b08fb34ad7ce3349ea989590c4830591fb8
See mtrskeen/ninfer-5080 for the build and serving instructions. Recommended 16 GB profile:
ninfer-serve qwen3_8_27b_minq4_mtpq4_visionq4.ninfer \
--model-id qwen3.8-27b-16gb \
--host 0.0.0.0 --port 18100 \
--max-context 124928 --kv-capacity 124928 \
--kv-dtype i4 --spec mtp --draft-tokens 3 \
--prefill-chunk 256 --no-cuda-graph
Vision is a separate mode: add --vision and cap context at 32,768 (the Vision
runtime has a hard 32,768 merged-token envelope, and --vision at 124,928 OOMs
at startup on 16 GB).
Baseline profile with --prefill-chunk 256 (validated, not the old 64):
| prompt tokens | prefill tok/s | TTFT | peak VRAM |
|---|---|---|---|
| 32,714 | 1,500 | 22 s | 15,858 MiB |
| 97,873 | 1,116 | 88 s | 15,858 MiB |
| 122,861 | 1,017 | 121 s | 15,858 MiB |
MTP3 decode is strongly content-dependent: ~144 tok/s on structured output
(97 % acceptance, 3.90 tokens/round), ~117–133 tok/s on code/translation,
~71–86 tok/s on story/long reasoning (31–43 % acceptance). MTP0 is a flat
~53 tok/s. gsm8k_cot (8-shot, lm-evaluation-harness, 50 samples): 0.96. RULER NIAH retrieval at 8K/32K/64K: 3/3 each.
Full methodology, per-fixture results, raw CSV/JSON and the context/OOM
boundaries are in the fork's BENCHMARKS.md.
sm_120a build. Other sm_120 16 GB cards are expected
compatible, not hardware-tested.--prefill-chunk 256 is a performance profile: it can change exact token
choices versus chunk 64 at near-ties (decode speed is unchanged), and it
lowers the absolute capacity boundary to 125,952 while the safe ceiling stays
124,928.--lm-head-draft does not fit at the 124,928 ceiling and needs a smaller
context.aljazceru/ninfer.mtrskeen/ninfer-5080.Licensed under Apache-2.0; see LICENSE.