Downloads · 30 days
1.2K
10% of all-time downloads
vladimir94/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-GGUF
Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-GGUF is a text generation model from vladimir94. Use it when you need the model to write or continue text. It is set up for llama.cpp. The card lists the license as apache-2.0.
GGUF conversion of YuYu1015/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4, itself an NVFP4 W4A4 quantization of huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated.
Downloads · 30 days
1.2K
10% of all-time downloads
All-time downloads
12.4K
Public
Repo size
27.1 GB
Likes
0
Public
Click a slice to open those files.
.gguf27.1 GB · 100%
From the Hugging Face model README
GGUF conversion of YuYu1015/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4, itself an NVFP4 W4A4 quantization of huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated.
This repository is intended for llama.cpp on NVIDIA Blackwell systems such as DGX Spark / GB10. The text model is provided as a main GGUF plus a separate MTP draft GGUF for llama.cpp speculative decoding. An optional mmproj file is also included for image input.
This is an uncensored/abliterated model. Use it responsibly.
| File | Size | Purpose |
|---|---|---|
Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-BF16.gguf | 20.95 GiB | Main text model, MTP excluded |
mtp-Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-BF16.gguf | 3.48 GiB | Separate MTP draft model for llama.cpp |
mmproj-Huihui-Qwen3.6-35B-A3B-abliterated-F16.gguf | 0.84 GiB | Optional vision projector |
The BF16 suffix means the converter used BF16 for non-NVFP4 tensors and fallback tensors. The source checkpoint is YuYu1015's NVFP4 packed model, and the llama.cpp runtime used for testing reported BLACKWELL_NATIVE_FP4 = 1.
Converted from the downloaded Hugging Face safetensors checkpoint with llama.cpp:
65ef50a0a4bb240211a41d43c957ae6313af6841version: 1 (65ef50a)BLACKWELL_NATIVE_FP4 = 1Main model and MTP were split deliberately:
--no-mtp was used for the main model.--mtp was used on a second pass to create the standalone draft GGUF.--model-draft.Equivalent commands:
SRC=/path/to/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4
OUT=/path/to/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-GGUF
LLAMA=/path/to/llama.cpp
python3 "$LLAMA/convert_hf_to_gguf.py" "$SRC" \
--outfile "$OUT/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-BF16.gguf" \
--outtype bf16 \
--no-mtp
python3 "$LLAMA/convert_hf_to_gguf.py" "$SRC" \
--outfile "$OUT/mtp-Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-BF16.gguf" \
--outtype bf16 \
--mtp
The vision projector needed a small conversion wrapper because this checkpoint stores visual tensors under model.language_model.visual.*, while the llama.cpp Qwen3-VL mmproj converter expects visual.*. The wrapper remaps only that prefix and then calls convert_hf_to_gguf.py.
Equivalent mmproj command after applying that prefix wrapper:
python3 convert_qwen35_mmproj.py "$SRC" \
--outfile "$OUT/mmproj-Huihui-Qwen3.6-35B-A3B-abliterated-F16.gguf" \
--outtype f16 \
--mmproj
Text-only, 262K context, MTP enabled:
llama-server \
--model Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-BF16.gguf \
--model-draft mtp-Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-BF16.gguf \
--alias huihui-qwen3.6-35b-a3b-abliterated-uncensored-nvfp4-mtp \
--ctx-size 262144 \
--parallel 8 \
--batch-size 8192 \
--ubatch-size 2048 \
--flash-attn on \
--n-gpu-layers all \
--kv-unified \
--cont-batching \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--cache-type-k-draft q8_0 \
--cache-type-v-draft q8_0 \
--spec-type draft-mtp \
--spec-draft-n-max 1 \
--draft-p-min 0.3 \
--cache-ram 8192 \
--reasoning off
For vision, add:
--mmproj mmproj-Huihui-Qwen3.6-35B-A3B-abliterated-F16.gguf
For the tested deployment, the non-vision preset used 8 slots. The vision preset used 2 slots, because loading mmproj adds memory and disables some cache-reuse behavior in llama.cpp multimodal mode.
All numbers below are llama.cpp generation throughput on DGX Spark / GB10 at 262K context with the text-only GGUF, --flash-attn on, Q8_0 KV cache, and native Blackwell FP4 support enabled. Continuous batching was enabled for the live llama-server tests and was verified with MTP. These are local smoke/throughput tests, not a formal benchmark suite.
| Setup | Throughput | Notes |
|---|---|---|
| No speculative decoding | ~31.0 tok/s | Baseline |
MTP, --spec-draft-n-max 1, --draft-p-min 0.0 | ~34.76 tok/s | Acceptance ~71.7% |
MTP, --spec-draft-n-max 1, --draft-p-min 0.3 | ~34.84 tok/s | Acceptance ~78.3%; selected setting |
MTP, --spec-draft-n-max 2, --draft-p-min 0.0 | ~33.99 tok/s | Slower than n=1 |
MTP, --spec-draft-n-max 2, --draft-p-min 0.3 | ~33.07 tok/s | Slower than n=1 |
MTP, --spec-draft-n-max 3, --draft-p-min 0.0 | ~31.88 tok/s | Close to baseline |
MTP, --spec-draft-n-max 4, --draft-p-min 0.0 | ~29.15 tok/s | Slower than baseline |
Recommended MTP setting for this GGUF:
--spec-type draft-mtp --spec-draft-n-max 1 --draft-p-min 0.3
Higher MTP depths were not useful in the local tests. We stopped the sweep after n=4 because throughput was already declining, and kept n=1.
DFlash was not benchmarked for this GGUF/llama.cpp release. The source NVFP4 model card discusses DFlash for vLLM, but this repository is focused on llama.cpp with the model's built-in MTP draft.
Tested on a DGX Spark / GX10 with 128GB unified memory.
One-slot temporary llama.cpp test:
n_max=1, draft_p_min=0.3Eight-slot live llama-server deployment:
--parallel 8, --ctx-size 262144, --kv-unified, --cont-batchingn_ctx = 262144The 8-slot throughput number is a single active request on an 8-slot server, not an 8-concurrent aggregate throughput benchmark.
Because --kv-unified is enabled, filling slots with context does not allocate eight separate full-size KV buffers. The live worker stayed around 30-31 GiB by nvidia-smi. Host memory can still grow due to prompt/checkpoint caching; the tested config set --cache-ram 8192.
The optional mmproj file was converted and a local image-input smoke test succeeded. The test image had a red/blue split, and the model correctly described red and blue side-by-side.
Recommended usage:
mmproj only when image input is needed.Known multimodal caveat from local llama.cpp logs:
mmproj loaded, llama.cpp reported that cache reuse is not supported by multimodal mode and disabled it.The tested llama.cpp deployment used:
--cache-ram 8192
--kv-unified
--cont-batching
Prompt/checkpoint caching worked in the sense that idle slots were saved and restored from the prompt cache. However, for this hybrid Qwen3.6/GDN context, llama.cpp also logged that cache_reuse was not supported and disabled cache reuse. In practice, do not expect every repeated prompt to show classic prefix-cache hit behavior.
This is an abliterated/uncensored model. It may produce unsafe, offensive, or policy-violating content. Users are responsible for deployment choices, filtering, logging, and compliance with applicable law.