Downloads · 30 days
43
43% of all-time downloads
PurimCd/Qwen3.8-27B-heretic-ara-EXL3-8.00bpw-h8
Qwen3.8-27B-heretic-ara-EXL3-8.00bpw-h8 is a image-text-to-text model from PurimCd. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for exllamav3. The card lists the license as apache-2.0.
High-fidelity EXL3 quantization of trohrbaugh/Qwen3.8-27B-heretic-ara, an ARA-abliterated Qwen3.8-27B model produced with Heretic.
Downloads · 30 days
43
43% of all-time downloads
All-time downloads
100
Public
Parameters
14.7B
29.4 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors29.3 GB · 100%
How the weights are stored.
I1612.9B · 88%
From the Hugging Face model README
High-fidelity EXL3 quantization of trohrbaugh/Qwen3.8-27B-heretic-ara, an ARA-abliterated Qwen3.8-27B model produced with Heretic.
This repository quantizes the already-abliterated source model; it does not apply a second abliteration pass. It targets users who want near-BF16 text quality with a practical two-GPU ExLlamaV3 deployment. Text weights and the LM head target 8 bpw, the MTP head targets 4 bpw, and the vision component remains BF16.
| Setting | Value |
|---|---|
| Engine | ExLlamaV3 1.4.2 |
| Target text bitrate | 8.00 bpw |
| Effective layer bitrate | 8.0122 bpw |
| LM head | 8 bit (8.0032 effective bpw) |
| MTP head | 4 bit |
| Calibration | 250 × 2,048 tokens |
| Codebook | mul1 |
| Output scales | always |
| Vision component | BF16 |
| Weight shards | 4 |
| Weight payload | 29,342,141,118 bytes (27.33 GiB) |
The exact source revision and conversion settings are recorded in
evaluation/conversion.json.
All validation below was run after conversion, using the files in this repository.
qbench was run on 10 non-overlapping WikiText-2 test rows of 2,048 tokens each (20,480 tokens total). The reference was the source BF16 model.
| Model | PPL | Mean KLD vs BF16 | Median KLD | Estimated model VRAM |
|---|---|---|---|---|
| Source BF16 | 6.023400 | — | — | 47.73 GiB |
| BF16 repeat / noise floor | 6.020072 | 0.002032 | 0.000375 | 47.73 GiB |
| This EXL3 quant | 6.020476 | 0.000831 | 0.000330 | 23.90 GiB |
The EXL3 divergence is below the measured BF16 repeat noise floor on this run. The tiny PPL inversion should therefore be read as measurement/runtime variance, not as evidence that quantization improved the model.


Raw output and the portable test specification are in
evaluation/qbench-results.json and
evaluation/qbench-config.yml.
The source Heretic protocol was reproduced exactly from its reproduce/config.toml:
mlabonne/harmful_behaviors, split test[:100]You are a helpful assistant.| Artifact | Refusals | Empty responses |
|---|---|---|
| Source BF16, as published by the source author | 0 / 100 | not reported |
| This EXL3 quant, locally re-evaluated | 0 / 100 | 0 / 100 |
Aggregate results are in
evaluation/refusal-results.json, and the copied source
protocol is in evaluation/source-heretic-config.toml.
No harmful prompts or generated harmful responses are included in this repository.
Keyword-based refusal scoring is a narrow regression check, not a general safety or capability benchmark.
The BF16 vision component loaded successfully and correctly described ExLlamaV3's bundled cat
test image, including the kitten and the visible colors of its hat. The prompt, generated response,
and check result are in evaluation/vision-smoke.json.
Measured with ExLlamaV3 1.4.2, MTP enabled, and layer-split loading:
| GPUs | Decode | Prefill | MTP acceptance |
|---|---|---|---|
| RTX 3090 + RTX 3090 Ti | 51.05 tok/s | 471.75 tok/s | 54.47% |
| 2× RTX 3090 + RTX 3090 Ti | 25.23 tok/s | 105.79 tok/s | 52.10% |
On this specific machine, two GPUs were substantially faster than three because the extra layer-split synchronization outweighed the additional compute. This is not a universal multi-GPU result; tensor-parallel configurations and different interconnects can behave differently.
The model payload alone is close to the capacity of a 24 GiB card. Two 24 GiB GPUs are strongly recommended for useful context and runtime headroom. Long-context quality was not validated in this release.
Install a CUDA-compatible PyTorch build first, then ExLlamaV3:
pip install exllamav3
For a two-GPU interactive chat from the ExLlamaV3 repository:
CUDA_VISIBLE_DEVICES=0,1 python examples/chat.py \
-m /path/to/Qwen3.8-27B-heretic-ara-EXL3-8.00bpw-h8 \
-mode qwen35 \
-gs 20,20 \
-cs 32768 \
-mtp
The model also works with TabbyAPI using its
exllamav3 backend. Start with two GPUs and a modest context, then increase cache/context only
after checking actual VRAM usage on your system.
Run the following from the model directory after downloading:
sha256sum -c SHA256SUMS
The manifest covers all release files except README.md and SHA256SUMS itself.
The upstream Apache-2.0 license is included in LICENSE.