Downloads · 30 days
126
16% of all-time downloads
XReyRobert/Qwen3.8-27B-GPTQ-Pro-FOEM-4bit-g128-ns256
Qwen3.8-27B-GPTQ-Pro-FOEM-4bit-g128-ns256 is a text generation model from XReyRobert. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This is a GPTQ-Pro FOEM quantization of Qwen/Qwen3.8-27B. It targets efficient single-GPU inference while preserving the base model's large token embedding, output projection, vision, MTP, and normalization tensors ou…
Downloads · 30 days
126
16% of all-time downloads
All-time downloads
807
Public
Parameters
27.8B
19.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors19.6 GB · 100%
How the weights are stored.
I3224.3B · 88%
From the Hugging Face model README
This is a GPTQ-Pro FOEM quantization of
Qwen/Qwen3.8-27B. It targets
efficient single-GPU inference while preserving the base model's large token
embedding, output projection, vision, MTP, and normalization tensors outside
the 4-bit transformer-body path.
The checkpoint retains the base model's native 262,144-token position limit. The maximum practical serving context is deployment-specific and depends on GPU memory, KV-cache precision, concurrency, CUDA-graph capture, and whether vision or speculative MTP decoding is enabled.
The quantization was produced from source revision
1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0.
| Setting | Value |
|---|---|
| Transformer body | GPTQ-Pro FOEM W4A16 |
| Weight bits | 4 |
| Group size | 128 |
| Symmetric | Yes |
| Activation ordering | Disabled (desc_act=false) |
| Sequential quantization | Enabled |
| Calibration samples | 256 |
| Calibration sequence length | 2048 |
| Calibration mix | Code and reasoning JSONL |
| Quantizer | GPTQModel 6.1.0-dev |
| Pack format | GPTQ, INT32 packing |
The recorded recipe uses activation-weighted MSE, act-group-aware
quantization, FOEM alpha=0.25 and beta=0.2, plus a 0.5% RTN fallback with
MSE smoothing. Full machine-readable settings are available in
quantize_config.json and config.json.
The following tensor families were excluded from the 4-bit GPTQ body and kept in their original precision:
lm_headThe repository includes model-preserved-skips.safetensors as an auxiliary
record of the 15 explicitly preserved tensors. The indexed checkpoint contains
2,399 weights totaling 19,559,449,368 bytes (approximately 18.22 GiB).
The following text-only profile was validated on one RTX 3090 24 GB. The
runtime was the local residency image
localhost/vllm-openai-nightly-7a1eb8ac2-sleepwake-local:20260617, based on
the pinned upstream vLLM nightly commit
7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8. Its local changes implement
sleep/wake residency control; they do not alter this checkpoint's
quantization format.
vllm serve XReyRobert/Qwen3.8-27B-GPTQ-Pro-FOEM-4bit-g128-ns256 \
--served-model-name qwen3.8-27b-gptq-pro-foem-4bit-g128-ns256-ctx110k \
--language-model-only \
--dtype float16 \
--quantization gptq_marlin \
--max-model-len 110592 \
--max-num-seqs 1 \
--kv-cache-dtype fp8_e5m2 \
--enable-prefix-caching \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--max-cudagraph-capture-size 32 \
--gpu-memory-utilization 0.95 \
--trust-remote-code
This validated profile does not use --enforce-eager. The 110,592-token
limit is a single-RTX-3090 deployment result, not a limitation encoded in the
checkpoint.
Vision and MTP weights are present, but enabling either feature changes the
memory budget and requires a separately sized serving profile. Remove
--language-model-only for vision use.
Use the source model's recommended sampling parameters:
| Mode | Temperature | Top-p | Top-k | Min-p | Presence penalty | Repetition penalty |
|---|---|---|---|---|---|---|
| Thinking | 1.0 | 0.95 | 20 | 0.0 | 0.0 | 1.0 |
| Instruct / non-thinking | 0.7 | 0.8 | 20 | 0.0 | 1.5 | 1.0 |
Qwen3.8 supports reasoning_effort=low, medium, or xhigh; xhigh is the
source-model default. Thinking is enabled by default. For agentic sessions,
retain prior reasoning blocks with:
{"enable_thinking": true, "preserve_thinking": true}
Pass that object as chat_template_kwargs. For direct non-thinking replies,
set enable_thinking=false and use the non-thinking sampling row above.
Large max_tokens values are intentional for long-horizon agentic tasks. The
source model recommends separate reasoning and final-response headroom when
the serving stack supports those limits. Keep input, retained reasoning, and
new output within the deployment's actual max_model_len.
For single-24-GB-GPU deployments requiring more KV-cache capacity, see
XReyRobert/Qwen3.8-27B-GPTQ-Pro-FOEM-4bit-g128-ns256-INT8-Head-Embeddings.
That derivative keeps this 4-bit GPTQ-Pro transformer body and additionally
quantizes the token embeddings and lm_head to group-size-128 INT8. It was
validated at 210K text-only context, 182K with vision, and 170K with MTP2 on
one RTX 3090 24 GB. It requires the narrow vLLM compatibility patch included
in that repository.
The hybrid derivative's published Terminal-Bench 2.0 Smoke24 campaign scored
16/24 with reasoning_effort=low and 17/24 with both medium and
xhigh, using one pass per effort on its 210K text profile. See the
hybrid model card
for the exact conditions and per-task artifacts. These results belong to the
hybrid INT8 derivative and must not be attributed to this non-hybrid
checkpoint.
max_model_len=110592 on one RTX 3090.