Downloads · 30 days
116
8% of all-time downloads
Zynerji/Qwen3.8-27B-PristinelyUncensored-HOMEUSER-16-24
Qwen3.8-27B-PristinelyUncensored-HOMEUSER-16-24 is a text generation model from Zynerji. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
A 27B hybrid-attention VLM with vision and speculative decoding intact, in 14.75 GB. Every number below was measured on that GPU, on rented hardware, with the flags shown further down. Nothing is extrapolated.
Downloads · 30 days
116
8% of all-time downloads
All-time downloads
1.4K
Public
Parameters
4.4B
14.7 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors14.7 GB · 100%
How the weights are stored.
I324.4B · 99%
From the Hugging Face model README
A 27B hybrid-attention VLM with vision and speculative decoding intact, in 14.75 GB. Every number below was measured on that GPU, on rented hardware, with the flags shown further down. Nothing is extrapolated.
| GPU | VRAM | max context | tok/s @4k | MTP k=2 | TTFT 3.5k |
|---|---|---|---|---|---|
| RTX 5090 | 32 GB | 262,144 | 87.6 | 0.92x | 0.98 s |
| RTX 4090 | 24 GB | 262,144 | 56.4 | 0.98x | 1.30 s |
| RTX 3090 Ti | 24 GB | 262,144 | 56.1 | 1.49x | 2.40 s |
| RTX 3090 | 24 GB | 262,144 | 50.4 | 1.11x | — |
| RTX A4000 | 16 GB | 14,336 | 28.3 | ✗ fails | 3.78 s |
| RTX 5060 Ti | 16 GB | 12,288 | 27.4 | ✗ fails | 4.49 s |
| RTX 5070 Ti | 16 GB | 10,240 | 50.8 | ✗ fails | 2.25 s |
| RTX 4080 Super | 16 GB | not cleanly measured | 43.6 | ✗ fails | 2.00 s |
✗ fails = the engine will not start with MTP enabled — no headroom for the draft model on 16 GB.
— = not measured. TTFT is time-to-first-token on a real 3,500-token prompt, not an empty one.
262,144 is the model's max_position_embeddings — on 24 GB and above there is no context left
to give, confirmed on four independent machines across three GPU generations.
The 16 GB ceiling is card-specific. Three cards whose visible VRAM differs by 0.16 GiB landed 4,096 tokens apart, so measure yours rather than trusting a single number — anything held by a display server or compositor comes straight off it. 16,384 fit on no 16 GB card tested.
This model runs on Zynerji/vllm, not on stock vLLM:
pip install git+https://github.com/Zynerji/vllm@ektome-homeuser-v2
Pinned commit: 29f0f1c61f4f4ddcb553de92e6893c9524a93b80
Why it is required. This is the only Qwen3.8-27B quant that compresses embed_tokens and
lm_head — two 248320 x 5120 tensors that every other release leaves in BF16. That is where the
~6 GB of headroom comes from, and it is why 262,144 tokens of context fit on a 24 GB card at all.
Stock vLLM cannot load them. qwen3_5.py constructs VocabParallelEmbedding without passing
quant_config, so a quantized embedding is built unquantized and load dies with
no module or parameter named embed_tokens.weight_packed. The identical omission appears a second
time in qwen3_5_mtp.py, so the MTP speculative head cannot start either
(... in Qwen3_5MultiTokenPredictor). llama.py already passes both arguments; the Qwen paths do
not. The fork fixes all three sites.
The change is small and unencumbered — anyone is welcome to lift it:
# vllm/model_executor/models/qwen3_5.py AND qwen3_5_mtp.py
self.embed_tokens = VocabParallelEmbedding(
self.vocab_size,
config.hidden_size,
quant_config=quant_config, # <-- add
prefix=maybe_prefix(prefix, "embed_tokens"), # <-- add
)
Two flags are load-bearing on every card. --max-num-batched-tokens 2048 is pure activation
memory (the 8192 default costs you 192k->256k on 24 GB and drops 16 GB to 4k), and
--gpu-memory-utilization must not exceed 0.97 — at 0.98 the engine dies with
EngineDeadError at contexts that succeed at 0.97, so utilization is non-monotonic here.
24 GB card — the long-context configuration:
vllm serve Zynerji/Qwen3.8-27B-PristinelyUncensored-HOMEUSER-16-24 \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.97 \
--max-num-batched-tokens 2048 \
--max-num-seqs 16
Every flag here is load-bearing. Drop --max-num-batched-tokens 2048 and the ceiling falls to
192k; raise utilization to 0.98 and the engine dies with EngineDeadError.
16 GB card — it fits, but only just:
vllm serve Zynerji/Qwen3.8-27B-PristinelyUncensored-HOMEUSER-16-24 \
--max-model-len 10240 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.97 \
--max-num-batched-tokens 2048 \
--max-num-seqs 1
--kv-cache-dtype fp8 and --max-num-batched-tokens 2048 are both mandatory on 16 GB — at the
8192 default the ceiling collapses to 4k.
The 16 GB ceiling is card-specific. Measure yours; do not assume 12,288. Cards whose visible VRAM differs by 0.16 GiB landed 4,096 tokens apart:
| card | VRAM visible to vLLM | max context | tok/s @4k | TTFT 3.5k |
|---|---|---|---|---|
| RTX A4000 | 15.63 GiB | 14,336 | 28.3 | 3.78 s |
| RTX 5060 Ti | 15.48 GiB | 12,288 | 27.4 | 4.49 s |
| RTX 5070 Ti | 15.47 GiB | 10,240 | 50.8 | 2.25 s |
Walk --max-model-len down from 16384 in 2048 steps until the engine starts; anything left over
by a display server or compositor comes straight off this number. 16,384 did not fit on any 16 GB
card tested. If you need long context, use a 24 GB card — the 16 GB tier is "a 27B that fits",
not a long-context machine.
Do not enable MTP on a 16 GB card. There is no headroom for the draft model at util 0.97 and the engine fails to start (verified on 5060 Ti, 5070 Ti, 4080 Super and A4000 — every 16 GB card tested, across two independent sweeps).
--max-num-seqs matters on this architecture. 48 of the 64 layers are linear-attention
(gated-delta) and each concurrent decode sequence consumes one Mamba state block. On 24 GB there
are ~39 blocks, and vLLM refuses to start above that:
ValueError: max_num_seqs (128) exceeds available Mamba cache blocks (39).
Concurrency is bounded by Mamba state, not by KV cache. Keep max_num_seqs <= 32 on 24 GB.
RTX 3090 (24 GB), vLLM 0.27.1, this exact checkpoint.
| Benchmark | Result | n |
|---|---|---|
| MMLU | 0.7995 | 1531 |
| Compliance (refusal rate) | 0.988 — 5 refusals | 416 |
| HumanEval pass@1 (by execution) | 0.8902 | 164 |
| MMStar (vision) | 0.6633 | 300 |
HumanEval and MMStar were measured on the identical checkpoint with a BF16 lm_head; the int4
lm_head shipped here scores MMLU 0.7995 vs that build's 0.7975 at the same n=1531 — a tie, since
an unpaired test at this n needs ~2.8 points to resolve.
| card | context | KV | VRAM | tok/s |
|---|---|---|---|---|
| 24 GB | 262,144 | fp8 | 22.97 / 23.56 GiB | 7.4 eager · 50.6 graphs |
| 24 GB | 2,048 | fp8 | 20.0 GiB | 50.3 (b1) / 341.9 (b8) |
| 16 GB | 10,240–14,336 | fp8 | ~15.3 / 15.5 GiB | 27–51 |
Use CUDA graphs. With enforce_eager the same 256k configuration runs at 8.5 tok/s; with
graphs enabled (the default) it runs at 50.6 tok/s — a 6x difference, and the single biggest
performance factor after the flags above.
Long context costs prefill, not decode. Measured time-to-first-token with the cache genuinely full: ~26 s at 30k, ~2.6 min at 125k, ~7.3 min at 250k (prefill throughput falls from ~1,150 to ~575 tok/s as the quadratic attention term grows). If you query a stable corpus repeatedly, enable prefix caching so that cost is paid once rather than per request.
Only the 16 full-attention layers grow with context — 64 KB/token bf16, 32 KB/token fp8. The other 48 layers hold constant-size state. That is why 256k costs ~8 GiB of KV instead of the ~100 GiB a dense 27B would need.
985 modules, 14.75 GB. Every group sits at the precision its runtime kernel can serve — the constraints are geometric, not stylistic.
| component | precision | why |
|---|---|---|
| decoder (658 Linears) | int4 g128, GPTQ-calibrated | MarlinLinearKernel; all 400 2-D decoder weights satisfy Marlin tiling |
| vision tower (110 Linears) | int4 g16, RTN | linear_fc2 has in_features 4304 = 16x269; 16 is the only group size under 128 that divides it, and only HummingLinearKernel accepts 16 |
embed_tokens | int4 g128 symmetric | vLLM's embedding path requires INT + symmetric |
lm_head | int4 g128 symmetric | needs symmetric and a per-group format: pack-quantized; asymmetric int4 finds no scheme |
mamba in_proj_a/b, conv1d | BF16, 137.6 MB | out_features=48; 48 % 64 != 0 fails Marlin tiling |
| MTP head (15 tensors) | FP8 + BF16 norms | round-trip verified, rel-L1 0.0224–0.0226 |
| quantization scales | BF16, 399.7 MB | irreducible metadata, 2.7% of the file |
lm_head quantisation error is rel-L1 0.1184, lower than the embedding that ships alongside it
(0.173).
The 15-tensor MTP head is retained and FP8-quantised rather than dropped:
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
Speculative decoding is exact at temperature 0 — drafts are verified against the target model, so this changes throughput and never quality.
MTP is card-dependent — benchmark it on your own GPU before leaving it on. Measured at 4k, batch 1, CUDA graphs on, across a 10-GPU sweep:
| GPU | no spec | MTP k=2 | gain |
|---|---|---|---|
| RTX 3090 Ti | 56.1 | 83.4 | 1.49x |
| RTX 3090 | 50.4 | 55.9 | 1.11x |
| RTX 4090 | 56.4 | 55.1 | 0.98x |
| RTX 5090 | 87.6 | 80.4 | 0.92x |
| any 16 GB card | — | — | fails to initialise |
Ampere gains; Ada and Blackwell lose. On 16 GB there is no headroom for the draft model at util 0.97 and the engine will not start with MTP enabled — leave it off there.
Gains also depend on how predictable the output is (structured text and code accept more drafts
than free prose) and shrink as concurrency rises. MTP requires the fork's qwen3_5_mtp patch;
without it the engine fails to start on any card.
The tower is retained and functional (MMStar 0.6633, vs 0.6500 for an FP8 tower on the same items —
int4-g16 costs nothing measurable). Pass enable_thinking=False for classification-style prompts:
this checkpoint's template injects a high reasoning-effort preamble, and without the flag the model
spends its token budget reasoning instead of answering.
From Zynerji/Ektome-Qwen3.8-27B-PristinelyUncensored (Ektomē refusal-direction excision), itself
from Qwen/Qwen3.8-27B. Structure verified complete against the parent: 985/985 modules.
--max-num-batched-tokens 2048; at the 8192 default the ceiling is 192k.embed_tokens fix.