Downloads · 30 days
90
7% of all-time downloads
Smayan/Qwen3.8-27B-Uncensored-Genesis-V1-MTP-NVFP4-GGUF
Qwen3.8-27B-Uncensored-Genesis-V1-MTP-NVFP4-GGUF is a text generation model from Smayan. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
An NVFP4 quantisation of LuffyTheFox/Qwen3.8-27B-Uncensored-Genesis-V1-MTP-GGUF, built to Luffy's requested per-tensor quant policy and verified running on an RTX 5070 Ti (Blackwell) with native FP4 support enabled.
Downloads · 30 days
90
7% of all-time downloads
All-time downloads
1.4K
Public
Repo size
15.8 GB
Likes
3
Public
Click a slice to open those files.
.gguf15.8 GB · 100%
From the Hugging Face model README
An NVFP4 quantisation of LuffyTheFox/Qwen3.8-27B-Uncensored-Genesis-V1-MTP-GGUF, built to Luffy's requested per-tensor quant policy and verified running on an RTX 5070 Ti (Blackwell) with native FP4 support enabled.
@LuffyTheFox — this is yours, take it. Mirror it into your repo, or say the word and I'll open a PR against it directly.
| Source | Qwen3.8-27B-Uncensored-Genesis-V1-Q8_K_P.gguf (29990 MiB, 9.21 BPW) |
| Output | Qwen3.8-27B-Uncensored-Genesis-V1-NVFP4.gguf (15024 MiB, 4.61 BPW) |
| Tensors | 502 NVFP4 · 4 F16 · 360 F32 |
| Fallbacks to another type | none — every eligible tensor took NVFP4 |
| Architecture | qwen35, 27.32 B params, 64 layers (+1 MTP/NextN head, preserved) |
| Context | 262144 |
Full offload on a 16 GB RTX 5070 Ti, llama.cpp build b10666:
system_info: CUDA : ARCHS = 750,800,860,890,900,1200,1210 | BLACKWELL_NATIVE_FP4 = 1
load_tensors: offloaded 66/66 layers to GPU
load_tensors: CPU_Mapped model buffer size = 682.03 MiB
load_tensors: CUDA0 model buffer size = 14114.16 MiB
ftype : NVFP4
At -c 4096, fully offloaded: ~285–340 t/s prompt, ~42–46 t/s generation.
Quality was checked by running the model's output, not reading it. Asked for
a thread-safe O(1) LRU cache with an explicit doubly-linked list and no
OrderedDict, the generated code passed: correct eviction order (evicting the
genuinely-least-recent node, not the just-touched one), the capacity-0 case,
ValueError on negative capacity, and 8 threads x 4000 operations with the
linked list still exactly in sync with the dict afterwards. A damaged quant does
not write a correct sentinel-node LRU with __slots__ and worked-out lock
granularity.
Luffy's policy was written for the MoE Qwen 3.6 35B, which has
ffn_down_exps. This model is dense, so that tensor does not exist here;
the equivalent is ffn_down. Applied that way, every line resolves:
| tensor | type | note |
|---|---|---|
blk.0.attn_gate.weight | F16 | as specified |
blk.0.attn_qkv.weight | F16 | as specified |
blk.0.ffn_down.weight | F16 | dense equivalent of ffn_down_exps |
blk.13.ffn_down.weight | F16 | " |
blk.0.ssm_out.weight | NVFP4 | as specified |
blk.1.attn_gate.weight | NVFP4 | as specified |
blk.1.attn_qkv.weight | NVFP4 | as specified |
| everything else eligible | NVFP4 | catch-all, listed last |
Verified in the finished file — all four F16 tensors are exactly the intended ones, and the three explicit NVFP4 tensors are NVFP4.
One caveat worth stating plainly: blk.0.attn_gate / attn_qkv were q8_0 in
the source, so F16 there stores dequantised q8 values. It does the intended
job (those tensors keep q8-level fidelity instead of dropping to 4 bits) but it
is not true F16 precision, and it costs more bytes than the source did.
The interesting part: no code changes and no rebuild were needed. A stock
llama-quantize can already do this.
llama-quantize does not expose NVFP4 as a base CLI ftype — QUANT_OPTIONS in
tools/quantize/quantize.cpp has no entry for it, and llama-quant.cpp has no
ftype→type case. The obvious read is that you must patch and rebuild. You don't,
because of three things in the existing code:
--tensor-type resolves its type argument through ggml_type_name() over
all ggml types, so =nvfp4 already parses.vector<pair<regex, ggml_type>> and is matched
first-match-wins, so a trailing .* catch-all is expressible.ggml_is_quantized(default_type)), so any quantised base works — the
catch-all then replaces it everywhere.So the base ftype is a placeholder that never survives, and the real policy is the regex list. Specific patterns first, catch-all last:
llama-quantize \
--allow-requantize \
--tensor-type "blk\.0\.attn_gate\.weight=f16" \
--tensor-type "blk\.0\.attn_qkv\.weight=f16" \
--tensor-type "blk\.0\.ffn_down\.weight=f16" \
--tensor-type "blk\.13\.ffn_down\.weight=f16" \
--tensor-type "blk\.0\.ssm_out\.weight=nvfp4" \
--tensor-type "blk\.1\.attn_gate\.weight=nvfp4" \
--tensor-type "blk\.1\.attn_qkv\.weight=nvfp4" \
--tensor-type ".*=nvfp4" \
Qwen3.8-27B-Uncensored-Genesis-V1-Q8_K_P.gguf \
Qwen3.8-27B-Uncensored-Genesis-V1-NVFP4.gguf \
Q8_0
Then the file type label, which is the one thing the above gets wrong. The
header inherits the placeholder base ftype and reports file type = Q8_0. That
is cosmetic — loading dispatches on per-tensor types — but it is wrong on the
shelf, so patch the single uint32 in place (no 15 GB rewrite):
python gguf-py/gguf/scripts/gguf_set_metadata.py model.gguf general.file_type 39 --force
39 is LLAMA_FTYPE_MOSTLY_NVFP4. The loader guesses the ftype correctly from
the majority tensor type on its own (GGML_TYPE_NVFP4 → LLAMA_FTYPE_MOSTLY_NVFP4
is already in that switch in llama-model-loader.cpp) and only overrides the
guess when general.file_type is present — so a stale value is actively worse
than none. After the patch it reads ftype : NVFP4.
--allow-requantize already dequantises each block to f32 before
quantising, so routing through F16 only adds a rounding step.--dry-run resolves the whole policy without writing anything, which is how
the missing ffn_down_exps was caught before spending any time.tensor_type_fallback handles tensors whose shape isn't divisible by the
NVFP4 block size, warning per tensor. Here it never fired.llama-cli -m Qwen3.8-27B-Uncensored-Genesis-V1-NVFP4.gguf \
-ngl 99 -c 4096 --temp 0.6 --top-k 20 --top-p 1.0 --min-p 0.0
Sampling above follows the Genesis guidance (temp 0.6, top_k 20, top_p and min_p
disabled). For code, also set --repeat-penalty 1.0: code repeats
identifiers by nature, and any penalty pushes the model off the identifier it
just chose, which is a reliable way to get generated code that won't run.
Genesis reasons at length before answering. In testing it spent 900 tokens,
then 2600 tokens, entirely inside [Start thinking] without reaching its final
answer. That is the model, not the quantisation — but with a small -n it looks
like the model is broken or rambling. Budget generously (-n 4096 or more) for
anything non-trivial, or you will only ever see it thinking.
Needs a Blackwell card and a llama.cpp built with FP4 support for the fast path
(BLACKWELL_NATIVE_FP4 = 1 in system_info); b10666 or newer.
Base model and quant policy: LuffyTheFox. Genesis V1 finetune: HauhauCS. This repo is only the NVFP4 conversion and the verification that it runs.