Downloads · 30 days
10
8% of all-time downloads
namgyu-youn/Qwen3-8B-W4A16-INT
Qwen3-8B-W4A16-INT is a text generation model from namgyu-youn. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
- Developed by: namgyu-youn - License: apache-2.0 - Quantized from Model: Qwen/Qwen3-8B - Quantization Method: W4A16-INT
Downloads · 30 days
10
8% of all-time downloads
All-time downloads
133
Public
Repo size
6.2 GB
Likes
0
Public
Click a slice to open those files.
.bin6.2 GB · 100%
From the Hugging Face model README
lm_eval --model hf --model_args pretrained=Qwen/Qwen3-8B --tasks mmlu --device cuda:0 --batch_size 8 --limit 100
| Groups | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| mmlu | 2 | none | acc | ↑ | 0.7542 | ± | 0.0055 | |
| - humanities | 2 | none | acc | ↑ | 0.7577 | ± | 0.0112 | |
| - other | 2 | none | acc | ↑ | 0.7408 | ± | 0.0116 | |
| - social sciences | 2 | none | acc | ↑ | 0.8333 | ± | 0.0105 | |
| - stem | 2 | none | acc | ↑ | 0.7111 | ± | 0.0101 |
lm_eval --model hf --model_args pretrained=namgyu-youn/Qwen3-8B-W4A16-INT --tasks mmlu --device cuda:0 --batch_size 8 --limit 100
| Benchmark | ||
|---|---|---|
| Qwen/Qwen3-8B | namgyu-youn/Qwen3-8B-W4A16-INT | |
| mmlu | - | - |
vllm bench throughput --model Qwen/Qwen3-8B --input-len 256 --output-len 256 --num-prompts 100
Throughput: 7.20 requests/s, 3684.21 total tokens/s, 1842.11 output tokens/s
vllm bench throughput --model namgyu-youn/Qwen3-8B-W4A16-INT --input-len 256 --output-len 256 --num-prompts 100
| Benchmark | ||
|---|---|---|
| Qwen/Qwen3-8B | namgyu-youn/Qwen3-8B-W4A16-INT | |
| Throughput (requests/s) | 7.2 | - |
vllm bench latency --model Qwen/Qwen3-8B --input-len 256 --output-len 256 --batch-size 1
Avg latency: 5.971084968124827 seconds
10% percentile latency: 5.963464400172233 seconds
25% percentile latency: 5.96858908329159 seconds
50% percentile latency: 5.972043054178357 seconds
75% percentile latency: 5.9746390748769045 seconds
90% percentile latency: 5.976582102105022 seconds
99% percentile latency: 5.97972804877907 seconds
vllm bench latency --model namgyu-youn/Qwen3-8B-W4A16-INT --input-len 256 --output-len 256 --batch-size 1
| Benchmark | ||
|---|---|---|
| Qwen/Qwen3-8B | namgyu-youn/Qwen3-8B-W4A16-INT | |
| Latency (ms) | 5.96 | - |