Downloads · 30 days
30
16% of all-time downloads
Qwen3-4B-Instruct-2507-FP8
Subalzt/Qwen3-4B-Instruct-2507-FP8
Qwen3-4B-Instruct-2507-FP8 is a text generation model from Subalzt. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
An FP8 (E4M3) dynamic quantization of Qwen/Qwen3-4B-Instruct-2507, produced with llm-compressor (FP8DYNAMIC, data-free). Runs on NVIDIA Blackwell's native FP8 tensor cores.
All-time downloads
183
Public
Parameters
4.4B
5.2 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors5.2 GB · 100%
Parameter types
How the weights are stored.
F8_E4M33.6B · 82%
Try a prompt
Base models
Model card
From the Hugging Face model README
Qwen3-4B-Instruct-2507 — FP8 (E4M3)
An FP8 (E4M3) dynamic quantization of
Qwen/Qwen3-4B-Instruct-2507,
produced with llm-compressor
(FP8_DYNAMIC, data-free). Runs on NVIDIA Blackwell's native FP8 tensor cores.
Quality — near-lossless (measured, not asserted)
wikitext-2-raw-v1, context 2048, 145 non-overlapping chunks, scored under transformers. Compared paired, per chunk (the correct test — the tiny error bars overlap but the paired statistic is decisive):
| model | perplexity | vs BF16 | size |
|---|---|---|---|
| BF16 (base) | 10.0216 | — | 7.5 GB |
| FP8 (this) | 10.0415 | +0.20 % (paired t = 5.4, p = 3e-7) | 4.85 GB |
| GPTQ W4A16 | 10.3261 | +3.04 % | 2.48 GB |
| AWQ W4A16 | 10.5907 | +5.68 % | 3.21 GB |
The +0.20 % is statistically real but negligible — FP8 beats 4-bit methods by an order of magnitude in quality at ~2× their size. 8-bit quantization is essentially free; the quality cost is all in the jump to 4-bit.
Speed & VRAM (RTX 5070 Laptop, 8 GB, sm_120)
| metric | value |
|---|---|
| VRAM to load | ~5.5 GB (incl. ~1.3 GB CUDA context) |
| weights on disk | 4.85 GB |
| decode speed | 47.8 tok/s (single stream, vLLM) |
FP8 is slower than the 4-bit quants (8-bit = more weight bytes read per token; decode is memory-bandwidth bound) but nearly lossless — the choice when quality matters and you have the VRAM.
Usage (vLLM)
from vllm import LLM
llm = LLM("Subalzt/Qwen3-4B-Instruct-2507-FP8")
print(llm.generate("Explain the TCP three-way handshake.")[0].outputs[0].text)
Provenance & reproducibility
Full method, code, and the comparison against GPTQ/AWQ/GGUF: github.com/Subalzt/qwen3-4b-quantized-fp8-gptq. Benchmarked on an RTX 5070 Laptop (8 GB, sm_120) under WSL2.
License
Apache-2.0, inherited from the base model. Attribution: Qwen team,
Qwen/Qwen3-4B-Instruct-2507.
- Task
- Text Generation
- Type
- qwen3
- License
- apache-2.0
- Languages
- en
- Created
- Aug 9, 2026
- Updated
- Aug 10, 2026