Downloads · 30 days
129
14% of all-time downloads
Lygodactylus/Qwen3.8-27B-Uncensored-exl3-6bpw
Qwen3.8-27B-Uncensored-exl3-6bpw is a text generation model from Lygodactylus. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
<p align="center" <img src="https://cdn-uploads.huggingface.co/production/uploads/67fcdd9530700089d649ed6e/xHG-9ipsnpjtplgIU2aWr.png" width="200" alt="Qwen3.8-27B-Uncensored EXL3" / </p <h1 align="center"Qwen3.8-27B-U…
Downloads · 30 days
129
14% of all-time downloads
All-time downloads
925
Public
Parameters
11.7B
23.5 GB on disk
Likes
4
Public
Click a slice to open those files.
.safetensors23.5 GB · 100%
How the weights are stored.
I1610B · 85%
From the Hugging Face model README
EXL3 quantization of orcarouter/Qwen3.8-27B-Uncensored, an abliterated (refusal-removed) build of Qwen/Qwen3.8-27B.
Built with ExLlamaV3 1.4.6.
22 GB on disk.
The MTP head is preserved at 8 bpw and works. GGUF conversions of this
family drop the mtp.* tensors, so they have no self-drafter. TabbyAPI logs
Using main model MTP component for drafting on load.
| Variant | Size | Repo |
|---|---|---|
| 4.0 bpw | 16 GB | Qwen3.8-27B-Uncensored-exl3-4bpw |
| 6.0 bpw | 22 GB | this one |
| 8.0 bpw | 28 GB | Qwen3.8-27B-Uncensored-exl3-8bpw |
| Language model | 6.00 bpw |
lm_head | 8 bpw |
| MTP layers | 8 bpw |
| Vision tower | unquantized (16-bit) |
| Embeddings | unquantized (16-bit) |
python convert.py \
-i Qwen3.8-27B-Uncensored \
-o Qwen3.8-27B-Uncensored-exl3-6bpw \
-w /tmp/exl3-work \
-b 6.0 -hb 8 -mb 8 -vb 16 \
-d 0,1,2,3 -v
Default bundled calibration corpus (wiki 50, C4 20, code 20, random tokens 20, technical 10, multilingual 10, tiny 5), 250 rows x 2048 columns.
model:
model_dir: /path/to/models
model_name: Qwen3.8-27B-Uncensored-exl3-6bpw
cache_size: 32768
cache_mode: "8,8"
tensor_parallel: true
draft_model:
draft_mode: mtp
Hardware: 4x RTX 4000 Ada (20 GiB, sm89), PCIe, no NVLink, 2x Xeon
Gold 5318Y. AIPerf, streaming, ignore_eos, seed 42, with warmup.
ExLlamaV3 1.4.6 + TabbyAPI, tensor-parallel, cache_mode "8,8", MTP on.
ISL 1000 / OSL 500 / concurrency 4 / 20 requests:
| 4.0 bpw | 6.0 bpw | 8.0 bpw | |
|---|---|---|---|
| Size | 16 GB | 22 GB | 28 GB |
| Request latency | 19,906 ms | 20,816 ms | 21,145 ms |
| ITL | 33.5 ms | 35.0 ms | 36.2 ms |
| TTFT | 3,352 ms | 3,515 ms | 3,289 ms |
| Output throughput | 97 tok/s | 91 tok/s | 90 tok/s |
75% more weight from 4 to 8 bpw costs only 6% latency. On four tensor-parallel cards over PCIe, weight bandwidth is not the bottleneck — inter-GPU synchronization dominates. Pick the bitrate on VRAM budget and quality, not on speed.
The 8 bpw TTFT being lowest of the three is noise; 20 requests can't resolve 5% gaps.
| vLLM 0.28.0 FP8 | EXL3 6.0 bpw | |
|---|---|---|
| Request latency | 14,158 ms | 20,816 ms |
| ITL | 26.3 ms | 35.0 ms |
| TTFT | 1,028 ms | 3,515 ms |
Long prompts (ISL 8000 / conc. 2): vLLM TTFT 5,508 ms vs 9,481 ms, prefill 1,733 vs 1,277 tok/s.
vLLM is faster in every served scenario on this box. EXL3 wins on VRAM and
on single-stream short prompts (63 vs ~45 tok/s in chat.py).
tensor_parallel is a trade, not a win| TP off | TP on | |
|---|---|---|
| Prefill TTFT (8k) | 17,644 ms | 9,577 ms |
| Single-stream generation | 63 tok/s | 34.6 tok/s |
Helps compute-bound prefill, hurts memory-bound decode. State which one you ran when quoting EXL3 numbers.
Caveat: PCIe, no NVLink, SymmMemCommunicator unavailable on sm89. TP
results are likely better on NVLink systems. Treat these as a floor.
Not benchmarked at 6.0 bpw. For reference on the same abliterated weights, the FP8 build scores 88.0% on MMLU-Pro (business subset, 100 questions, 2 unparsed) vs 89.0% for official Qwen3.8-27B-FP8 — within noise. How 6.0 bpw compares has not been verified.
Inherited from the base model: safety alignment has been substantially removed via abliteration. It will comply with requests the original Qwen3.8-27B refuses, and has no meaningful built-in guardrails. Released for research and controlled experimentation. Add your own moderation layer before any deployment. You are responsible for what you do with it.
Apache 2.0, inherited from Qwen/Qwen3.8-27B.