Downloads · 30 days
19
16% of all-time downloads
Qwen3.5-4B-fp8
mahadev9/Qwen3.5-4B-fp8
Qwen3.5-4B-fp8 is a machine learning model from mahadev9. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
FP8 (W8A8 dynamic) quantization of Qwen/Qwen3.5-4B, produced with llm-compressor. lmhead, mlp.downproj, selfattn.oproj, and the first/last transformer layers are kept in the original precision - the layers most consis…
All-time downloads
116
Public
Parameters
4.2B
16.4 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors5.8 GB · 100%
Parameter types
How the weights are stored.
F8_E4M32.6B · 61%
Base models
Model card
From the Hugging Face model README
Qwen3.5-4B-fp8
FP8 (W8A8 dynamic) quantization of Qwen/Qwen3.5-4B,
produced with llm-compressor.
lm_head, mlp.down_proj, self_attn.o_proj, and the first/last
transformer layers are kept in the original precision - the layers most
consistently found to be quantization-sensitive - to recover
agentic/tool-calling quality.
Quality
KL divergence of the quantized model's next-token distribution vs the bf16
baseline (seq len 2048). prose is WikiText-2; agentic is tool-calling
turns (schema + query + call) rendered with the chat template.
| bucket | tokens | mean KL | top-1 agreement |
|---|---|---|---|
| prose | 24,130 | 0.0054 | 96.15% |
| agentic | 80,808 | 0.2057 | 91.92% |
| overall | 104,938 | 0.1596 | 92.89% |
Usage
vLLM:
vllm serve mahadev9/Qwen3.5-4B-fp8
transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"mahadev9/Qwen3.5-4B-fp8", torch_dtype="auto", device_map="auto"
)
tok = AutoTokenizer.from_pretrained("mahadev9/Qwen3.5-4B-fp8")
- Type
- qwen3_5_text
- Created
- Aug 30, 2026
- Updated
- Sep 5, 2026