Downloads · 30 days
73
9% of all-time downloads
P4pps3n/Qwen3.8-27B-exl3-4.0bpw-hb8-single3090
Qwen3.8-27B-exl3-4.0bpw-hb8-single3090 is a text generation model from P4pps3n. Use it when you need the model to write or continue text. It is set up for exllamav3. The card lists the license as apache-2.0.
This repository contains a local EXL3 quantization of Qwen/Qwen3.8-27B for ExLlamaV3.
Downloads · 30 days
73
9% of all-time downloads
All-time downloads
772
Public
Parameters
8.7B
17.4 GB on disk
Likes
3
Public
Click a slice to open those files.
.safetensors17.4 GB · 100%
How the weights are stored.
I166.9B · 80%
From the Hugging Face model README
This repository contains a local EXL3 quantization of Qwen/Qwen3.8-27B for ExLlamaV3.
The reduced calibration set was required because larger calibration allocations exceeded the available host memory during conversion. Benchmark this artifact before using it as a production replacement for higher-calibration quantizations.
Measured with ExLlamaV3 on one RTX 3090, one request per fresh process, temperature 0, top-p 1, and 127 generated tokens per run. The reported rate is decode throughput, not prompt prefill throughput.
| Prompt context | Prefill seconds | Decode tok/s |
|---|---|---|
| 128 | 0.941 | 43.19 |
| 2,048 | 1.868 | 41.37 |
| 8,192 | 8.594 | 40.30 |
| 32,768 | 32.546 | 36.96 |
| 65,536 | 76.878 | 33.28 |
The 128-token run used ExLlamaV3's minimum 2048-token cache workspace. The 64K run is a measured prefill and generation test, not a claim that the model fits at the native 256K limit on a 24 GB card.
The current deployment uses TabbyAPI with ExLlamaV3 on one RTX 3090. The RTX 3090 is not used as a desktop/display GPU on the test system.
max_seq_len: 196608
cache_size: 196608
cache_mode: Q8
max_batch_size: 1
chunk_size: 2048
draft_mode: mtp
The model loaded successfully with n_ctx: 196608. Observed NVIDIA memory after loading was 22,011 MiB used and 2,114 MiB free out of 24,576 MiB. GPU 0 in the server process is the RTX 3090 selected by CUDA_VISIBLE_DEVICES, not the RTX 5070 Ti.
These measurements were made against the running TabbyAPI service after the Q8/196,608-token configuration was loaded. Temperature was 0, thinking was disabled, one request was active, and the completion was limited to 64 tokens. Prompt token counts are the actual TabbyAPI counts.
| Prompt context | Prefill tok/s | Decode tok/s | Generation |
|---|---|---|---|
| 130,012 | 624.52 | 30.13 | 20 tokens |
| 190,012 | 368.16 | 23.87 | 20 tokens |
The server is configured with max_batch_size: 1; parallel requests are queued rather than decoded concurrently. Long-context throughput varies with prompt content, cache reuse, draft acceptance, and system load.
The following additional values are taken from current TabbyAPI generation logs during normal Hermes requests using MTP drafting. These are observational production measurements rather than isolated controlled benchmark runs.
| Context | Prefill tok/s | Decode tok/s | Draft acceptance |
|---|---|---|---|
| 70,578 | 323.94 | 46.03 | 837 / 1,743 |
| 72,184 | 438.62 | 47.23 | 1,157 / 2,300 |
| 74,213 | 351.18 | 64.95 | 932 / 1,147 |
| 76,599 | 509.54 | 32.13 | 989 / 3,722 |
| 80,739 | 502.59 | 48.45 | 803 / 1,445 |
| 81,585 | 272.33 | 43.24 | 882 / 1,830 |
The latest complete entry was 81,585 context tokens at 43.24 decode tok/s. Recent observed decode speed ranged from 32.13 to 64.95 tok/s, primarily due to variable MTP draft acceptance and request shape.
Apache-2.0, subject to the base model license and terms.