AxionML-Qwen3.5-9B-NVFP4-GGUF

AxionML Qwen3.5-9B-NVFP4 GGUF

This repository contains a GGUF conversion of AxionML/Qwen3.5-9B-NVFP4 for llama.cpp-compatible runtimes such as LM Studio.

Files

• AxionML-Qwen3.5-9B-NVFP4.gguf — main model
• mmproj-BF16.gguf — multimodal projector for image support

Notes

• The main GGUF was converted from the Hugging Face NVFP4 checkpoint using convert_hf_to_gguf.py
• Conversion was performed with --outtype bf16, producing a mixed-format GGUF with preserved supported tensor types and required floating-point auxiliary tensors
• Conversion command used:

python .\convert_hf_to_gguf.py `
  .\original-NVFP4 `
  --outfile .\converted-NVFP4.gguf `
  --outtype bf16 `
  --verbose

• The mmproj-BF16.gguf file was sourced from unsloth/Qwen3.5-9B-GGUF and verified working in local testing
• Original source model: AxionML/Qwen3.5-9B-NVFP4
• Base model family: Qwen/Qwen3.5-9B
• This 9B release follows the same general conversion idea as the earlier 27B NVFP4 GGUF build, but is aimed much more naturally at lower-VRAM RTX 50 series cards

What this build actually is

The upstream AxionML checkpoint is an NVFP4-quantized model created with NVIDIA TensorRT Model Optimizer. In other words, the source checkpoint is not a normal BF16 checkpoint and not a standard llama.cpp-style Q4/Q5 quant.

At the same time, the command for convert_hf_to_gguf.py uses --outtype bf16. In current llama.cpp conversion logic, that does not automatically mean “every tensor becomes plain BF16.” For ModelOpt/NVFP4 checkpoints, the converter contains dedicated handling for NVFP4 tensors, repacks them into GGUF NVFP4, and may also emit small floating-point auxiliary tensors such as scale tensors where required by the runtime: NVFP4 tensors are repacked and stored as GGMLQuantizationType.NVFP4, while separate floating-point scale tensors may also be written where needed. In addition, some very small tensors may remain FP32 regardless of --outtype, because certain operations require it.

this repository stores a mixed-format GGUF generated from an AxionML NVFP4 upstream checkpoint, where the main quantized tensors remain NVFP4-aware in GGUF form, while some auxiliary tensors remain floating-point where needed.

That is why the command line and the final tensor storage can look contradictory at first glance even though both are technically correct.

Why RTX 50 series matters here

NVFP4 is a Blackwell-era FP4 format. That matters because GeForce RTX 50 series GPUs are Blackwell GPUs, and Blackwell introduces hardware support for FP4 / NVFP4-style workflows.

So in practice:

• RTX 50 series is the most natural target for this kind of build
• When the backend/runtime actually uses the relevant FP4 path well, RTX 50 series can benefit more directly
• On RTX 40 / 30 / 20 series, a model derived from NVFP4 may still load or run depending on the runtime path, but those older generations do not have Blackwell’s native FP4 support, so the upside is less about true native FP4 acceleration and more about compatibility, footprint, or whatever fallback path the runtime uses

In short: this kind of build makes the most sense on RTX 50 series, but the exact speedup still depends on the current state of LM Studio, llama.cpp, CUDA kernels, batch settings, and context size, etc.

📊 Qwen 3.5 vs Qwen 3 Benchmark Overview

Higher is better.
This repository is based on Qwen3.5-9B, one of the strongest small-to-mid-size balanced models in the Qwen family.

[!NOTE] Benchmark note: The comparison table above is a community summary / visualization, with the visual overview based on this Reddit post: Visualizing all Qwen 3.5 vs Qwen 3 benchmarks

For official benchmark details from the Qwen team, see the benchmark section of: Qwen/Qwen3.5-9B

Quantized checkpoints may show small quality differences relative to their base model on benchmark results.

✨ Why Qwen3.5-9B stands out

Qwen3.5-9B looks like a very practical sweet spot for local RTX 50 series use when 27B is too large for the intended VRAM budget.

Key highlights from the comparison above:

• Strong knowledge / STEM performance for its size
• Very solid instruction-following and general agent capability
• Good long-context behavior compared with smaller local models
• Strong math and multilingual balance in a package that is much easier to place on 12GB–16GB cards than 27B

In short, Qwen3.5-9B is a compelling local-use model for users who want a strong all-rounder without stepping up to 27B-class VRAM pressure.

🚀 Practical local comfort zone

If the 27B NVFP4 GGUF was the comfortable-zone build for an RTX 5090 32GB, this 9B NVFP4 GGUF is the more natural sweet spot for smaller RTX 50 series VRAM tiers.

Based on local LM Studio Windows testing / placement guidance:

• RTX 50 series 16GB VRAM: comfortable zone at about 120K context
• RTX 50 series 12GB VRAM: comfortable zone at about 45K context

That makes this build especially attractive for:

• long-document reading on smaller cards
• code assistance without jumping to a 27B-class memory footprint
• practical local multimodal use with a projector file
• extended chat and retrieval-heavy workflows on mainstream Blackwell GPUs

Performance note: actual throughput and max usable context will vary depending on LM Studio version, backend/kernel support, prompt shape, batch settings, GPU memory reservation, and whether image support is enabled.

About the earlier 27B build

For anyone who saw the earlier AxionML-Qwen3.5-27B-NVFP4-GGUF release and is wondering whether this 9B build is “the same idea”: yes.

The earlier 27B release was likewise built from AxionML’s upstream NVFP4 checkpoint family and converted into a GGUF suitable for local runtimes. This 9B build follows the same overall concept, just at a model size that is more realistic for 12GB–16GB RTX 50 series local use.

🔥 At a glance

Qwen3.5-9B combines:

• strong all-round capability for its size
• much better local practicality than 27B on lower VRAM tiers
• a Blackwell-relevant NVFP4 upstream source
• good long-context usability for 12GB–16GB RTX 50 series targets
• a cleaner local sweet spot for users who want strong quality without flagship-class memory demands

That makes this GGUF release a particularly strong option for users who want a model that is both high quality for its class and practical to run locally.

License

Please refer to the upstream model license and attribution requirements.