Downloads · 30 days
91
1% of all-time downloads
olka-fi/Qwen3.5-122B-A10B-MXFP4
Qwen3.5-122B-A10B-MXFP4 is a text generation model from olka-fi. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
MXFP4 quantized version of Qwen3.5-122B-A10B (122B parameters, 256 experts, hybrid Gated DeltaNet + Gated Attention).
Downloads · 30 days
91
1% of all-time downloads
All-time downloads
11.5K
Public
Parameters
70.5B
79.2 GB on disk
Likes
11
Public
Click a slice to open those files.
.safetensors79.2 GB · 100%
How the weights are stored.
U861.8B · 88%
From the Hugging Face model README
MXFP4 quantized version of Qwen3.5-122B-A10B (122B parameters, 256 experts, hybrid Gated DeltaNet + Gated Attention).
Expert MLP weights only are quantized to MXFP4 (4-bit microscaling with e8m0 shared exponents, block size 32). All attention, linear attention (Gated DeltaNet), gates, embeddings, and normalization layers remain in BF16.
| Original (BF16) | This model (MXFP4) | |
|---|---|---|
| Size on disk | 250 GB | 74 GB |
| Perplexity (wikitext, 2048 ctx) | — | 4.90 |
| Component | Precision | Notes |
|---|---|---|
| Expert gate_up_proj, down_proj | MXFP4 (uint8 packed + e8m0 scales) | 3D fused expert tensors [256, out, in] |
| Shared expert gate/up/down_proj | MXFP4 | 2D standard linear weights |
| Self-attention (Q/K/V/O projections) | BF16 | Excluded — preserves attention quality |
| Linear attention (Gated DeltaNet layers) | BF16 | Excluded |
| MoE router (mlp.gate) | BF16 | Excluded — routing must be exact |
| Shared expert gate | BF16 | Excluded |
| Embeddings, LM head | BF16 | Excluded |
| Visual encoder | BF16 | Excluded |
| MTP layers | BF16 | Excluded |
| LayerNorm weights | BF16 | 1D, not quantizable |
pip install vllm
vllm serve /path/to/Qwen3.5-122B-A10B-MXFP4 \
--quantization compressed-tensors \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.95 \
--max-model-len 24576 \
--max-num-batched-tokens 4096 \
--enable-prefix-caching
Minimum VRAM: ~114 GB (e.g., 2x A100-80GB with TP=2, or 4x RTX Pro 6000 with TP=4).
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
response = client.chat.completions.create(
model="Qwen3.5-122B-A10B-MXFP4",
messages=[{"role": "user", "content": "Who are you?"}],
)
print(response.choices[0].message.content)
*self_attn*, *linear_attn*, *.mlp.gate, *shared_expert_gate*, *lm_head*, *embed_tokens*, *visual*, *mtp*Perplexity measured on wikitext with sliding window:
Windows: 578 max_len=2048 stride=512
Tokens scored: 297,471
Mean NLL: 1.5899
Perplexity: 4.9032
seq_len (14) < num_heads (16) — false positive from the Flash Linear Attention ops during short warmup sequences. Does not affect output quality.Based on Qwen3.5-122B-A10B by Tongyi Lab (Alibaba).