Downloads · 30 days
22
19% of all-time downloads
Qwen3.5-0.8B-fp8
mahadev9/Qwen3.5-0.8B-fp8
Qwen3.5-0.8B-fp8 is a machine learning model from mahadev9. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
FP8 (W8A8 dynamic) quantization of Qwen/Qwen3.5-0.8B, produced with llm-compressor. lmhead, mlp.downproj, selfattn.oproj, and the first/last transformer layers are kept in the original precision - the layers most cons…
All-time downloads
115
Public
Parameters
752M
3.3 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors1.1 GB · 98%
Parameter types
How the weights are stored.
BF16386M · 51%
Base models
Model card
From the Hugging Face model README
Qwen3.5-0.8B-fp8
FP8 (W8A8 dynamic) quantization of Qwen/Qwen3.5-0.8B,
produced with llm-compressor.
lm_head, mlp.down_proj, self_attn.o_proj, and the first/last
transformer layers are kept in the original precision - the layers most
consistently found to be quantization-sensitive - to recover
agentic/tool-calling quality.
Quality
KL divergence of the quantized model's next-token distribution vs the bf16
baseline (seq len 2048). prose is WikiText-2; agentic is tool-calling
turns (schema + query + call) rendered with the chat template.
| bucket | tokens | mean KL | top-1 agreement |
|---|---|---|---|
| prose | 24,130 | 0.0056 | 95.32% |
| agentic | 80,808 | 0.3003 | 88.16% |
| overall | 104,938 | 0.2325 | 89.81% |
Usage
vLLM:
vllm serve mahadev9/Qwen3.5-0.8B-fp8
transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"mahadev9/Qwen3.5-0.8B-fp8", torch_dtype="auto", device_map="auto"
)
tok = AutoTokenizer.from_pretrained("mahadev9/Qwen3.5-0.8B-fp8")
- Type
- qwen3_5_text
- Created
- Aug 30, 2026
- Updated
- Sep 5, 2026