Downloads · 30 days
24
0% of all-time downloads
williamliao/Qwen3.6-27B-Unsloth-NVFP4-Experimental-GGUF
Qwen3.6-27B-Unsloth-NVFP4-Experimental-GGUF is a text generation model from williamliao. Use it when you need the model to write or continue text. It is set up for llama.cpp. The card lists the license as apache-2.0.
[!WARNING] Experimental community conversion This repository contains an experimental GGUF conversion of unsloth/Qwen3.6-27B-NVFP4. Upstream llama.cpp added compressed-tensors NVFP4 conversion support in PR 21095. How…
Downloads · 30 days
24
0% of all-time downloads
All-time downloads
5K
Public
Repo size
23.2 GB
Likes
1
Public
Click a slice to open those files.
.gguf23.2 GB · 100%
From the Hugging Face model README
[!WARNING] Experimental community conversion
This repository contains an experimental GGUF conversion of
unsloth/Qwen3.6-27B-NVFP4.Upstream llama.cpp added compressed-tensors NVFP4 conversion support in PR #21095. However, this source checkpoint uses a mixed-precision configuration containing multiple quantization groups, including NVFP4-packed weights, FP8 tensors, and unquantized MTP tensors.
This specific GGUF was produced with a local experimental extension for that mixed layout. It has been tested only with the hardware and llama.cpp builds listed below. Compatibility, numerical behavior, and performance may differ across converter revisions, backends, devices, and future llama.cpp versions. Verify output quality before relying on it.
Current local testing does not show a clear generation-speed advantage over Q4_K_M. Treat this repository primarily as an experimental compatibility and research conversion rather than a performance-oriented replacement for mature K-quant GGUF files.
This is a community GGUF conversion for use with llama.cpp.
It is not a fully NVFP4 model. The resulting GGUF uses a mixed layout:
NVFP4Q8_0unsloth/Qwen3.6-27B-NVFP4Qwen/Qwen3.6-27BNVFP4 + Q8_0 + BF16The source checkpoint uses mixed-precision compressed-tensors groups:
lm_headFor the GGUF uploaded to this repository, the historical local converter extension handled the stored layout as follows:
Q8_0 with --fp8-as-q8Two separate llama.cpp changes are relevant:
These changes solve different parts of the pipeline:
compressed-tensors NVFP4 checkpoint
|
| PR #21095: conversion support
v
NVFP4 GGUF
|
| PR #22196: Blackwell CUDA kernels
v
llama.cpp runtime
Support for loading or executing NVFP4 does not guarantee that every operator, tensor shape, or token-generation path uses the fastest available FP4 Tensor Core kernel. End-to-end performance still depends on tensor type, operator shape, kernel dispatch, batch size, device placement, and the llama.cpp revision.
The model loads and runs with a recent native sm_120a llama.cpp CUDA build.
On the tested RTX 5070 Ti + RTX 5060 Ti dual-GPU system, the conversion is
functional and performs within a few percent of the tested NVIDIA
Qwen3.6-27B-NVFP4-GGUF.
The current mixed NVFP4 GGUF does not provide a clear generation-speed advantage over the mature Q4_K_M path.
Latest mixed-task results:
This conversion is currently most useful for:
compressed-tensors conversion researchThis conversion is experimental and is not an official Unsloth, Qwen, NVIDIA, or llama.cpp release.
Known limitations:
Q8_0, not preserved as native FP8sm_120a code does not prove that every NVFP4 operation uses an
optimal FP4 Tensor Core kernelA recent llama.cpp build with support for the following is required:
Tested with:
b9967-4f37f5197sm_120a--tensor-split 14,12--spec-draft-n-max 4--spec-draft-p-min 0.60The static CUDA build was inspected with cuobjdump and contained native
sm_120a cubins. This confirms that the benchmark did not rely only on older
architecture cubins or PTX fallback.
Native sm_120a code targets the correct GPU architecture, but it does not
prove that every NVFP4 operation uses an optimal FP4 Tensor Core kernel.
Runtime performance still depends on tensor type, operator shape, kernel
dispatch, device placement, and llama.cpp implementation details.
The embedded MTP tensors are usable, but on the tested dual-GPU system an external Q4_K_M MTP-only drafter was substantially faster while preserving almost the same aggregate acceptance rate.
llama-server.exe ^
-m "unsloth-Qwen3.6-27B-NVFP4.gguf" ^
-md "Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf" ^
--device-draft CUDA1 ^
--host 127.0.0.1 ^
--port 18010 ^
-np 1 ^
--threads 12 ^
--threads-batch 16 ^
--split-mode layer ^
--tensor-split 14,12 ^
--ctx-size 81920 ^
--no-mmap ^
-ngl -1 ^
--flash-attn on ^
--jinja ^
--ubatch-size 512 ^
--batch-size 2048 ^
--no-warmup ^
--temp 0.6 ^
--top-k 20 ^
--top-p 0.80 ^
--min-p 0.0 ^
--presence-penalty 1.5 ^
--repeat-penalty 1.0 ^
--fit off ^
--reasoning off ^
--spec-type draft-mtp ^
--spec-draft-n-max 4 ^
--spec-draft-p-min 0.60
llama-server.exe ^
-m "unsloth-Qwen3.6-27B-NVFP4.gguf" ^
--spec-type draft-mtp ^
--spec-draft-n-max 4 ^
--spec-draft-p-min 0.60 ^
--flash-attn on ^
-ngl -1
The embedded path may consume more bandwidth because the MTP tensors remain at higher precision. Performance also depends on how llama.cpp places the embedded MTP tensors across multiple GPUs.
At the time this GGUF was created, the upstream llama.cpp converter stopped on this checkpoint with:
NotImplementedError: Can't handle multiple config groups for compressed-tensors yet
Upstream llama.cpp later added compressed-tensors NVFP4 conversion support in PR #21095. That support normalizes compressed-tensors NVFP4 metadata and scales into the existing ModelOpt-compatible conversion path.
This checkpoint additionally contains a mixed FP8 + NVFP4 configuration and unquantized MTP tensors. The historical local extension used for the uploaded file specifically handled that mixed layout:
nvfp4-pack-quantized tensors were exported as native GGUF
NVFP4Q8_0 with
--fp8-as-q8Users converting the model again should use a recent upstream llama.cpp revision and verify the resulting tensor layout. Exact output may differ from the historical local conversion used to produce this repository.
Example conversion command:
python convert_hf_to_gguf.py \
/path/to/unsloth-Qwen3.6-27B-NVFP4 \
--outfile unsloth-Qwen3.6-27B-NVFP4.gguf \
--outtype auto \
--verbose
Expected mixed layout:
NVFP4-packed compatible tensors -> native GGUF NVFP4
FP8 tensors -> dequantized and stored as Q8_0
embedded MTP tensors -> BF16 / source floating-point precision
No neural-network weights were retrained.
Inspect the result with a recent llama-gguf build:
.\llama-gguf.exe .\unsloth-Qwen3.6-27B-NVFP4.gguf r
Expected tensor examples:
tensor: name = blk.0.ffn_down.weight, type = nvfp4
tensor: name = blk.0.ffn_down.scale, type = f32
tensor: name = blk.0.ffn_down.input_scale, type = f32
The file should also contain Q8_0 tensors corresponding to source FP8
weights and floating-point tensors corresponding to the embedded MTP layers.
The earlier server-based mtp-bench workload primarily compared embedded and
external MTP execution on this experimental GGUF.
| Main model | Drafter | Draft precision | Acceptance | Wall time | Aggregate throughput |
|---|---|---|---|---|---|
| Unsloth experimental GGUF | embedded MTP | BF16 / source precision | 87.37% | 57.46 s | 32.77 tok/s |
| Unsloth experimental GGUF | external MTP-only | Q4_K_M | 86.88% | 42.12 s | 44.71 tok/s |
| NVIDIA NVFP4 GGUF comparison | native MTP | source precision | 76.55% | 46.00 s | 40.37 tok/s |
Aggregate throughput is total predicted tokens divided by total wall time.
The NVIDIA row is only a local comparison. Different conversion layouts, quantization choices, embedded MTP weights, and runtime behavior prevent it from being a controlled model-quality comparison.
code_python pred= 192 draft= 161 acc= 142 rate=0.882 tok/s=56.7
code_cpp pred= 53 draft= 47 acc= 41 rate=0.872 tok/s=59.8
explain_concept pred= 192 draft= 134 acc= 103 rate=0.769 tok/s=34.0
summarize pred= 48 draft= 42 acc= 33 rate=0.786 tok/s=47.3
qa_factual pred= 192 draft= 154 acc= 127 rate=0.825 tok/s=44.4
translation pred= 17 draft= 17 acc= 13 rate=0.765 tok/s=40.3
creative_short pred= 37 draft= 19 acc= 18 rate=0.947 tok/s=29.0
stepwise_math pred= 192 draft= 158 acc= 140 rate=0.886 tok/s=54.5
json_output pred= 192 draft= 155 acc= 149 rate=0.961 tok/s=67.8
long_reasoning pred= 192 draft= 155 acc= 132 rate=0.852 tok/s=47.9
repeat_pattern pred= 192 draft= 152 acc= 152 rate=1.000 tok/s=73.8
code_completion pred= 192 draft= 166 acc= 145 rate=0.874 tok/s=62.4
long_code_review pred= 192 draft= 141 acc= 109 rate=0.773 tok/s=35.7
Aggregate:
requests: 13
predicted tokens: 1883
draft tokens: 1501
accepted tokens: 1304
acceptance rate: 0.8688
total wall time: 42.12 s
code_python pred= 192 draft= 159 acc= 140 rate=0.880 tok/s=37.9
code_cpp pred= 53 draft= 47 acc= 41 rate=0.872 tok/s=39.8
explain_concept pred= 192 draft= 126 acc= 100 rate=0.794 tok/s=26.5
summarize pred= 48 draft= 41 acc= 33 rate=0.805 tok/s=34.0
qa_factual pred= 192 draft= 157 acc= 130 rate=0.828 tok/s=32.7
translation pred= 17 draft= 18 acc= 13 rate=0.722 tok/s=28.1
creative_short pred= 37 draft= 20 acc= 19 rate=0.950 tok/s=25.0
stepwise_math pred= 192 draft= 157 acc= 140 rate=0.892 tok/s=37.9
json_output pred= 192 draft= 155 acc= 149 rate=0.961 tok/s=45.5
long_reasoning pred= 192 draft= 158 acc= 132 rate=0.835 tok/s=34.4
repeat_pattern pred= 192 draft= 155 acc= 152 rate=0.981 tok/s=48.5
code_completion pred= 192 draft= 159 acc= 149 rate=0.937 tok/s=44.0
long_code_review pred= 192 draft= 145 acc= 110 rate=0.759 tok/s=28.0
Aggregate:
requests: 13
predicted tokens: 1883
draft tokens: 1497
accepted tokens: 1308
acceptance rate: 0.8737
total wall time: 57.46 s
sm_120a Mixed-Task ComparisonA second benchmark used a locally compiled native sm_120a build:
build: b9967-4f37f5197
main GPU: NVIDIA GeForce RTX 5070 Ti 16 GB
secondary GPU: NVIDIA GeForce RTX 5060 Ti 16 GB
split mode: layer
tensor split: 14,12
external draft model: Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf
draft device: CUDA1
speculative decoding: draft-mtp
draft max: 4
draft probability minimum: 0.60
temperature: 0
context size: 2048
maximum generated tokens: 1024
The workload covered translation, tagging, styled generation, Objective-C, LiveKit, tool-use prompting, KL divergence, Swift concurrency, release-note translation, and knowledge-graph planning.
| Main model | Prompt throughput | Generation throughput |
|---|---|---|
| Q4_K_M heretic comparison | 241.85 tok/s | 40.94 tok/s |
| NVIDIA Qwen3.6-27B NVFP4 GGUF | 227.58 tok/s | 38.85 tok/s |
| Unsloth experimental mixed NVFP4 GGUF | 220.06 tok/s | 37.71 tok/s |
Relative results:
These results apply only to this dual-GPU configuration and should not be generalized to other GPUs, single-GPU execution, larger batches, or different context lengths.
| Prompt category | Q4_K_M | NVIDIA NVFP4 | Unsloth mixed NVFP4 |
|---|---|---|---|
| Japanese translation | 28.0 | 24.6 | 25.2 |
| Danbooru tags | 42.6 | 46.8 | 47.7 |
| Styled weather sentence | 32.8 | 30.1 | 27.3 |
| Objective-C | 47.2 | 47.2 | 45.6 |
| LiveKit explanation | 35.5 | 32.2 | 30.5 |
| Tool-use prompt | 40.1 | 36.3 | 36.6 |
| KL divergence | 43.0 | 37.2 | 36.6 |
| Swift concurrency | 50.2 | 48.1 | 42.2 |
| Release note | 49.9 | 47.7 | 48.0 |
| Knowledge-graph planning | 40.1 | 38.3 | 37.4 |
Values are generation tokens per second.
Q4_K_M was faster in most tasks, while the NVFP4 models occasionally matched or exceeded it on individual prompts. Short single-prompt results are sensitive to output length, token distribution, speculative-decoding acceptance, and runtime scheduling.
The native sm_120a results show that missing native Blackwell cubins were not
the only cause of the earlier NVFP4 performance.
Even with verified native sm_120a code:
Likely contributing factors include:
Native sm_120a support improves compatibility and targets the correct
architecture, but it does not guarantee an end-to-end NVFP4 speedup.
The mixed-task outputs did not show a clear overall quality advantage for any NVFP4 GGUF.
The NVIDIA and experimental Unsloth NVFP4 outputs were generally in the same quality range:
The Q4_K_M comparison model also produced both strong answers and confident technical or factual errors.
This is not a controlled quantization-quality comparison because the tested files do not all use identical model weights:
uncensored-heretic-v2 variantA valid quality comparison would require:
Treat this benchmark primarily as a local runtime and practical-output comparison, not proof that one quantization format has better model quality.
On the tested system:
The results suggest that an external low-precision MTP-only draft model can be faster than embedded higher-precision MTP tensors on this dual-GPU setup.
They do not prove that precision alone is responsible. Device placement, synchronization, graph construction, memory bandwidth, and backend behavior may also contribute.
In the native sm_120a mixed-task test:
The NVIDIA NVFP4 GGUF was modestly faster than this conversion, indicating that the experimental conversion is reasonably close to the comparison implementation.
Neither NVFP4 model provided a clear end-to-end generation advantage over Q4_K_M on this setup. At the tested llama.cpp revision, Q4_K_M remains the more practical choice when maximum interactive generation speed is the priority.
Performance is highly dependent on:
The MTP benchmark does not include a non-speculative baseline and should not be interpreted as a universal MTP speedup ratio.
The ten-prompt CLI comparison was run once per prompt. It is useful for practical A/B testing but is not statistically rigorous.
Short responses are particularly sensitive to:
For regression testing, run each prompt at least three times and compare median generation throughput. Prompt throughput, generation throughput, and end-to-end output throughput should be treated as separate metrics.
This conversion demonstrates that the mixed FP8 + NVFP4
compressed-tensors checkpoint can be exported into a loadable GGUF while
preserving compatible tensors as native NVFP4.
Upstream llama.cpp now supports compressed-tensors NVFP4 conversion through PR #21095, while PR #22196 provides native Blackwell CUDA execution for applicable NVFP4 matrix operations. Those additions do not make this uploaded mixed-layout GGUF fully NVFP4, nor do they guarantee that batch-1 generation uses the fastest possible FP4 path for every operation.
The model is functional and performs within a few percent of the tested NVIDIA NVFP4 GGUF comparison.
Current native sm_120a llama.cpp testing does not show a practical
generation-speed advantage over Q4_K_M. On the tested dual-GPU system,
Q4_K_M remained approximately 5-9% faster in average generation throughput.
The most effective optimization observed was not the main weight format, but replacing the embedded higher-precision MTP tensors with an external Q4_K_M MTP-only draft model.
Future llama.cpp improvements to small-batch and token-generation NVFP4 kernels, operator coverage, dispatch behavior, converter support, and multi-GPU execution may change these results.
The source model is distributed under the Apache License 2.0.
Review the upstream unsloth/Qwen3.6-27B-NVFP4 and
Qwen/Qwen3.6-27B model cards before redistribution or commercial use.
This repository is an unofficial, experimental community conversion and is not endorsed by Unsloth, Qwen, NVIDIA, or the llama.cpp project.