Downloads · 30 days
193
18% of all-time downloads
zrlu/Huihui-Qwen3.8-27B-abliterated-GPTQ-Int4-sym-G128-MTP-BF16-B70
Huihui-Qwen3.8-27B-abliterated-GPTQ-Int4-sym-G128-MTP-BF16-B70 is a text generation model from zrlu. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
GPTQ-INT4 quantization, purpose-built and tested for single Intel Arc Pro B70 (Xe2, Battlemage) inference via vLLM XPU on Windows Docker Desktop + WSL2 — and for pi/opencode agent workloads on the same card.
Downloads · 30 days
193
18% of all-time downloads
All-time downloads
1.1K
Public
Parameters
27.8B
19.6 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors19.6 GB · 100%
How the weights are stored.
I3224.3B · 88%
From the Hugging Face model README
GPTQ-INT4 quantization, purpose-built and tested for single Intel Arc Pro B70 (Xe2, Battlemage) inference via vLLM XPU on Windows Docker Desktop + WSL2 — and for pi/opencode agent workloads on the same card.
DRAFT_INT4=0); see the deploy repository's README
("Runtime patches") for the shape mismatch behind that.Apache-2.0 — inherited unchanged from the base model. See LICENSE and the
base model card. This is a weight-only quantization; no rights are widened.
gptqmodel==7.3.2; run on a CUDA GPU (RTX 5090) with lazy checkpoint
loading; calibration: wikitext-2 (or bundled fallback text).bits=4, group_size=128, desc_act=false, sym=true, lm_head=false, dynamic={"-:.*mtp.*":{}} (MTP tensors stay BF16).Qwen3_5ForConditionalGeneration, image_token_id 248056.mtp.* tensors.Served with vLLM XPU 0.28.0 (kernels 0.1.12.3), MTP3 speculative decoding
with BF16 draft (DRAFT_INT4=0), prefix caching, qwen3_xml tool-call parser, fp8 KV,
KV cache 4.3 GiB (KV_CACHE_MEMORY_BYTES=4617089843), max-num-seqs=1.
No repetition/presence penalty — sampling is pure Qwen defaults from the
model's generation_config.json (temperature 1.0 / top_k 20 / top_p 0.95);
the deploy repo's patch_tile_mask.py boot patch covers the vLLM 0.28.0
!-degeneration bug instead. Benchmark/report details live in the
deployment repository
(benchmarks/ section).
One-click container: zrlu/qwen38-27b-arc-pro-b70:latest on Docker Hub
(entrypoint auto-downloads this repo to /model on first start; switch to any
other published B70 quant with -e HF_REPO=zrlu/<repo>), Dockerfile and
pi-agent setup in the deploy repository.
# standalone / eval
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("zrlu/Huihui-Qwen3.8-27B-abliterated-GPTQ-Int4-sym-G128-MTP-BF16-B70")
m = AutoModelForCausalLM.from_pretrained(
"zrlu/Huihui-Qwen3.8-27B-abliterated-GPTQ-Int4-sym-G128-MTP-BF16-B70",
device_map="auto")
Signed: zrlu. GPU: Arc Pro B70 (0xe223) + RTX 5090 (quantization only).