Downloads · 30 days
309
2% of all-time downloads
knoopx/Qwen3.6-35B-A3B-NVFP4-GGUF
Qwen3.6-35B-A3B-NVFP4-GGUF is a text generation model from knoopx. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
GGUF conversion of RedHatAI/Qwen3.6-35B-A3B-NVFP4 for use with llama.cpp and compatible runtimes.
Downloads · 30 days
309
2% of all-time downloads
All-time downloads
13.9K
Public
Repo size
45.9 GB
Likes
12
Public
Click a slice to open those files.
.gguf23.4 GB · 100%
From the Hugging Face model README
GGUF conversion of RedHatAI/Qwen3.6-35B-A3B-NVFP4 for use with llama.cpp and compatible runtimes.
| Property | Value |
|---|---|
| Base model | Qwen3.6-35B-A3B |
| Source checkpoint | RedHatAI/Qwen3.6-35B-A3B-NVFP4 |
| Quantization | NVFP4 (compressed-tensors) |
| Format | GGUF (21 GB, 1293 tensors) |
| Hidden size | 2048 |
| Layers | 40 (Mixture-of-Experts) |
| Total experts | 256 |
| Experts per token | 8 |
| Expert FFN size | 512 |
| Shared expert FFN size | 512 |
| Context length | 262,144 |
| Vocabulary size | 248,320 |
| Tokenizer | Qwen3 (BPE, qwen35 pre-tokenizer) |
Qwen3.6-35B-A3B is a mixture-of-experts multimodal model with:
Linear weight layers, preserving auxiliary tensors (norms, SSM convolutions, router gates) in BF16/F32Clone llama.cpp with NVFP4 support:
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git checkout b8967
Version b8967 (or later) includes PR #21095 for converting HuggingFace NVFP4 models quantized with compressed-tensors, plus native Blackwell NVFP4 CUDA support via PR #22196. The converter maps compressed-tensors tensor names (e.g., weight_global_scale) to the ModelOpt equivalents and adjusts scale values to their reciprocals — weights themselves need no adaptation.
Install conversion dependencies (Python 3.12 or 3.13):
pip install -r requirements/requirements-convert_hf_to_gguf.txt --no-cache-dir
Download the source model:
huggingface-cli download RedHatAI/Qwen3.6-35B-A3B-NVFP4 --local-dir ./model
Apply tokenizer compatibility patch — the upstream tokenizer_config.json uses TokenizersBackend which is not recognized by the converter:
sed -i 's/"TokenizersBackend"/"Qwen2Tokenizer"/g' ./model/tokenizer_config.json
Register the BPE pre-tokenizer hash — the Qwen3.6 tokenizer produces an unrecognized hash in get_vocab_base_pre(). Add this entry to convert_hf_to_gguf.py:
if chkhsh == "1444df51289cfa8063b96f0e62b1125440111bc79a52003ea14b6eac7016fd5f":
# ref: https://huggingface.co/RedHatAI/Qwen3.6-35B-A3B-NVFP4
res = "qwen35"
Run the conversion:
python convert_hf_to_gguf.py ./model \
--outfile Qwen3.6-35B-A3B-NVFP4.gguf \
--outtype bf16 \
--verbose
--outtype bf16 flag does not produce a standard BF16 model. NVFP4-compatible tensors are repacked into the GGUF where supported by the converter and runtime. Auxiliary tensors (layer norms, SSM parameters, router gates) remain in floating-point format.1444df51...) was added to get_vocab_base_pre() mapping to the qwen35 tokenizer preset, matching the \p{M} (combining marks) regex used by Qwen3 models.Benchmarks run using llama-bench from the llama.cpp Docker image (fc2b0053f / 8967) on CUDA 13 with a single GPU. Each model was tested at context lengths 2048, 4096, and 8192 tokens (prompt processing and 32-token token generation).
| Quantization | Size | Params | PP 2048 t/s | PP 4096 t/s | PP 8192 t/s | TG 32 t/s |
|---|---|---|---|---|---|---|
| NVFP4 | 20.94 GB | 34.66B | 9767 ± 67 | 9572 ± 16 | 9107 ± 14 | 136 ± 19 |
| Q4_K_XL | 20.81 GB | 34.66B | 9121 ± 33 | 8889 ± 47 | 8464 ± 48 | 188 ± 23 |
| Quantization | Size | Params | PP 2048 t/s | PP 4096 t/s | PP 8192 t/s | TG 32 t/s |
|---|---|---|---|---|---|---|
| NVFP4 | 17.50 GB | 26.90B | 5168 ± 28 | 5020 ± 4 | 4687 ± 6 | 63 ± 5 |
| Q4_K_XL | 16.39 GB | 26.90B | 3720 ± 10 | 3627 ± 6 | 3419 ± 13 | 66 ± 4 |
podman run --rm --name llama-bench-<id> \
--device=nvidia.com/gpu=all \
--security-opt label=disable \
-v ~/.cache/huggingface/:/root/.cache/huggingface \
ghcr.io/ggml-org/llama.cpp:full-cuda13-b8967 \
--bench -hf <model> -n 3 --n-prompt <context> --n-gpu-layers 999
Full benchmark script: benchmark.sh
This model includes a bundled multimodal projector (mmproj-BF16.gguf) for vision tasks. It is shared with the base Qwen3.6-35B-A3B architecture and sourced from Unsloth's GGUF conversion.
When using --hf, llama.cpp loads it automatically. For local usage, specify it with:
./llama-cli -m Qwen3.6-35B-A3B-NVFP4.gguf --mmproj mmproj-BF16.gguf -n 256 --prompt "Hello"
# Local usage (specify mmproj for vision)
./llama-cli -m Qwen3.6-35B-A3B-NVFP4.gguf --mmproj mmproj-BF16.gguf -n 256 --prompt "Hello"
# Via HuggingFace (auto-discovers mmproj)
./llama-cli --hf knoopx/Qwen3.6-35B-A3B-NVFP4-GGUF -n 256 --prompt "Hello"
The original Qwen3.6 model is licensed under the Apache 2.0 License. This GGUF conversion is a format transformation of that checkpoint and inherits the same license. See RedHatAI/Qwen3.6-35B-A3B-NVFP4 for full licensing details.