Downloads · 30 days
150
14% of all-time downloads
sheppo/Qwen3.8-27B-Heretic-Abliterated-W4A16-A100
Qwen3.8-27B-Heretic-Abliterated-W4A16-A100 is a text generation model from sheppo. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
W4A16 AutoRound checkpoint derived from the exact RVN BF16 GGUF and tested on one A100-SXM4-80GB. It measured 73.15 tok/s raw and 138.38 tok/s with the companion DFlash2 runtime on the frozen sampled-prose benchmark,…
Downloads · 30 days
150
14% of all-time downloads
All-time downloads
1K
Public
Parameters
5.8B
17.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors17.7 GB · 100%
How the weights are stored.
I3224.3B · 90%
From the Hugging Face model README
W4A16 AutoRound checkpoint derived from the exact RVN BF16 GGUF and tested on one A100-SXM4-80GB. It measured 73.15 tok/s raw and 138.38 tok/s with the companion DFlash2 runtime on the frozen sampled-prose benchmark, a 1.89x speedup.
This Hugging Face repo contains the W4A16 target. The accelerated runtime,
DFlash2 drafter, optional local vision assembly, exact launch settings, and
base-Qwen A100 results live in
elsheppo/qwen38-a100-fastpath.
This is a text-only, vLLM-compatible W4A16 AutoRound derivative of the exact
RVN-BF16.gguf file from
0bserverx/Qwen3.8-27B-Heretic-Abliterated-Uncensored-GGUF.
8581a3e4cd8cdeca9bb6709d81ebcbcdfc93fe43RVN-BF16.gguf53,808,272,896 bytesfe3cb9c7d067f0016fb8ab0150e2b226c4dfb4e63b497e46ada1e7744b035cdc96ce448039b3c36fa879b9f4c740a8ee50c0f9ba17,702,015,479 bytesThat final line matters. The convenient base-Qwen overlay replaces tensors. Applying it here could erase the RVN modifications we were trying to preserve.
The first attempt matched all 851 tensor names and shapes, loaded in vLLM, and still produced corrupted multilingual symbols.
The failure was upstream of quantization: the Qwen3.8 GGUF converter changes
several tensor representations. The corrected bridge inverted the non-GDN norm
shift, continuous-time A_log transform, and Gated DeltaNet value-head layout
before AutoRound saw the model. It then mapped all 851 source and target tensors,
checked that every value was finite, and produced the expected deterministic
BF16 answer before quantization resumed.
The user-facing model is Qwen3.8-27B. Qwen3_5ForCausalLM labels inside the
config and runtime are the architecture names used by the official release.
The primary cell used one A100-SXM4-80GB, concurrency one, temperature 0.8, top-p 0.95, top-k 20, thinking off, and a fixed eight-prompt prose workload.
| Serving mode | Median committed tok/s | Mean committed tok/s | Median TTFT |
|---|---|---|---|
| Raw RVN W4A16 | 73.15 | 73.31 | 132 ms |
| RVN + DFlash2, k=7 | 138.38 | 148.89 | 155 ms |
The median speedup is 1.89x. We had frozen a 140 tok/s target before the
run; this landed 1.62 tok/s short, so the experiment record says
promising-below-target.
A separate greedy story fixture measured 152.30 tok/s at C1 and 609.87 tok/s aggregate at C8. Different workload, different measurement. The primary prose number stays 138.38.
The checkpoint passed:
Claude Fable judged all 36 pairs without knowing which side was raw or DFlash2, then repeated the judgments after every A/B position was swapped. The same content side won 35 of 36 calls across the swap. After unblinding, the preferred responses were nearly evenly split between serving modes: raw led by two pairs in the first presentation and three in the swapped presentation.
We observed no material writing-quality regression from the tested DFlash2 path. That conclusion belongs to this sampling profile and this bounded battery.
git clone https://github.com/elsheppo/qwen38-a100-fastpath
cd qwen38-a100-fastpath
./scripts/bootstrap.sh rvn
./scripts/serve.sh rvn dflash2
./scripts/smoke.sh
The fast profile uses
syvai/Qwen3.8-27B-DFlash2-W4A16
at revision 4d30ec736ffc6b8688dc2ae2b502d9b48bdec279, seven draft
tokens, lookup drafting disabled, and a fixed 8 GiB state/KV pool.
Run the target without DFlash2 for comparison:
./scripts/serve.sh rvn raw
The model artifact on this page is still text-only. The companion repository can combine these exact text weights with a pinned 921 MB Qwen vision payload without replacing the RVN language tensors:
./scripts/bootstrap.sh rvn-vision
./scripts/serve.sh rvn-vision dflash2
./scripts/vision_smoke.sh
The assembled target measured 72.58 tok/s raw and 140.84 tok/s with DFlash2 on the frozen image-conditioned generation benchmark, a 1.94x speedup. In a separate 100-image breadth battery, the DFlash path scored 88/100 with the correct images and 35/100 after the images were deliberately mismatched.
rvn serving profile is text-only and launches with
--language-model-only. The optional rvn-vision profile assembles a
separate local conditional-generation target.The underlying work comes from Qwen, the RVN/Heretic source authors,
AutoRound, vLLM, the DFlash2 authors, and the syv-ai serving project. The
companion repository preserves exact revisions and links in its NOTICE.
Apache-2.0, following the Qwen base model and the source model card.