Downloads · 30 days
672
11% of all-time downloads
FreedomAISVR/Qwen3.6-35B-A3B-MXFP4-MOE-GGUF
Qwen3.6-35B-A3B-MXFP4-MOE-GGUF is a text generation model from FreedomAISVR. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
MXFP4MOE GGUF quantization of Qwen3.6-35B-A3B, Alibaba Cloud's efficient 35B MoE multimodal foundation model (3B activated) with 262K context, 256 experts, and Gated DeltaNet hybrid attention.
Downloads · 30 days
672
11% of all-time downloads
All-time downloads
6.1K
Public
Repo size
20.7 GB
Likes
5
Trending 1
Click a slice to open those files.
.gguf20.7 GB · 100%
From the Hugging Face model README
MXFP4_MOE GGUF quantization of Qwen3.6-35B-A3B, Alibaba Cloud's efficient 35B MoE multimodal foundation model (3B activated) with 262K context, 256 experts, and Gated DeltaNet hybrid attention.
MXFP4 (Microscaling FP4) is a 4-bit floating-point quantization format standardized by the Open Compute Project (OCP) in 2024, backed by AMD, NVIDIA, Microsoft, Meta, and OpenAI. It uses an E2M1 layout (1 sign, 2 exponent, 1 mantissa bit) with block-wise shared 8-bit scaling factors (block size 32), preserving dynamic range far beyond integer 4-bit formats.
In llama.cpp, MXFP4_MOE applies MXFP4 quantization specifically to MoE expert weights, while keeping router and shared expert layers in Q8_0 for precision where it matters most.
| Filename | Type | Size | Description |
|---|---|---|---|
qwen3.6-35b-a3b-mxfp4_moe.gguf | MXFP4_MOE | 18.4 GB | Quantized MoE text model weights |
qwen3.6-35b-a3b-mxfp4-moe-mmproj-f16.gguf | F16 mmproj | 899 MB | Vision encoder projector |
| Property | Value |
|---|---|
| Format | MXFP4_MOE (OCP MXFP4 experts + Q8_0 router/shared) |
| Element type | E2M1 (1 sign, 2 exponent, 1 mantissa) |
| Block size | 32 |
| Shared scale | 8-bit per block |
| Effective BPW | 4.57 |
| VRAM required (text) | ~18 GB |
| VRAM required (text + vision + 32K ctx) | ~20 GB |
Qwen3.6-35B-A3B features:
# Text + Image
llama-cli -m qwen3.6-35b-a3b-mxfp4_moe.gguf --mmproj qwen3.6-35b-a3b-mxfp4-moe-mmproj-f16.gguf --image photo.jpg -p "Describe this image in detail"
# Text only
llama-cli -m qwen3.6-35b-a3b-mxfp4_moe.gguf -p "Explain quantum computing in simple terms" -n 512
# OpenAI-compatible server
llama-server -m qwen3.6-35b-a3b-mxfp4_moe.gguf --mmproj qwen3.6-35b-a3b-mxfp4-moe-mmproj-f16.gguf --port 8080
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="FreedomAISVR/Qwen3.6-35B-A3B-MXFP4-MOE-GGUF",
filename="qwen3.6-35b-a3b-mxfp4_moe.gguf",
n_gpu_layers=-1,
)
response = llm.create_chat_completion([
{"role": "user", "content": "What is the capital of France?"}
])
print(response["choices"][0]["message"]["content"])
from huggingface_hub import hf_hub_download
for filename in ["qwen3.6-35b-a3b-mxfp4_moe.gguf", "qwen3.6-35b-a3b-mxfp4-moe-mmproj-f16.gguf"]:
hf_hub_download(
repo_id="FreedomAISVR/Qwen3.6-35B-A3B-MXFP4-MOE-GGUF",
filename=filename,
local_dir="./models"
)
1. Download source weights
huggingface_hub.snapshot_download("Qwen/Qwen3.6-35B-A3B")
2. Convert text model to F16 GGUF
convert_hf_to_gguf.py --outtype f16
3. Extract vision encoder
convert_hf_to_gguf.py --mmproj --outtype f16
4. Quantize to MXFP4_MOE
llama-quantize qwen3.6-35b-a3b-f16.gguf qwen3.6-35b-a3b-mxfp4_moe.gguf MXFP4_MOE
| Component | Specification |
|---|---|
| GPU | NVIDIA GeForce RTX 5060 Ti (Blackwell SM120) |
| VRAM | 16 GB GDDR7 |
| System RAM | 64 GB DDR4 |
The original Qwen3.6-35B-A3B is released under Apache 2.0. These quantized weights inherit that license.