Downloads · 30 days
133
1% of all-time downloads
Qwopus3.6-27B-v2-GPTQ-Pro-v1
XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1
Qwopus3.6-27B-v2-GPTQ-Pro-v1 is a text generation model from XReyRobert. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
All-time downloads
15.8K
Public
Parameters
27.4B
18.7 GB on disk
Likes
8
Public
Repo makeup
Click a slice to open those files.
.safetensors18.7 GB · 100%
Parameter types
How the weights are stored.
I3224.3B · 89%
Try a prompt
Base models
Model card
From the Hugging Face model README
Qwopus3.6-27B-v2 GPTQ-Pro FOEM 4-bit g128 ns256 v2
<p style="margin: 10px 0 6px 0; color: #475569; font-size: 14px; line-height: 1.5;"> These models are built and maintained on rented GPU compute. If you want to show some appreciation, a follow on X or a coffee helps keep the releases coming. </p> <div style="display: flex; flex-wrap: wrap; align-items: center; gap: 10px; margin: 8px 0 18px 0;"> <a href="https://x.com/xreyrobert" target="_blank" rel="noopener noreferrer" class="follow-link" style="display: inline-flex; align-items: center; justify-content: center; gap: 8px; min-height: 36px; box-sizing: border-box; padding: 9px 14px; border: 1px solid #111827; border-radius: 999px; background: #111827; color: #ffffff; text-decoration: none; font-weight: 700; font-size: 14px; line-height: 1;"> <svg viewBox="0 0 24 24" fill="currentColor" aria-hidden="true" style="width: 16px; height: 16px; flex: 0 0 auto;"><path d="M18.244 2.25h3.308l-7.227 8.26 8.502 11.24H16.17l-5.214-6.817L4.99 21.75H1.68l7.73-8.835L1.254 2.25H8.08l4.713 6.231zm-1.161 17.52h1.833L7.084 4.126H5.117z"></path></svg> Follow @xreyrobert </a> <a href="https://donate.stripe.com/bJe5kDbkwaYn3Jq9uL18c00" target="_blank" rel="noopener noreferrer" class="support-link" style="display: inline-flex; align-items: center; justify-content: center; gap: 8px; min-height: 36px; box-sizing: border-box; padding: 9px 14px; border: 1px solid #f59e0b; border-radius: 999px; background: #fbbf24; color: #111827; text-decoration: none; font-weight: 700; font-size: 14px; line-height: 1;"> Support this work </a> </div>This is a GPTQ-Pro 4-bit quantization of Jackrong/Qwopus3.6-27B-v2, built to make this excellent Qwopus/Qwen3.6 model practical to run in vLLM with GPTQ-Marlin kernels and long-context inference.
The goal is simple: preserve as much of the original model's character and capability as possible while making it efficient enough for single-GPU RTX 3090-class vLLM deployments.
This is not a new fine-tune. It is a quantized derivative of the original Qwopus3.6-27B-v2 model.
Source and credits
Source model:
Quantization methodology and reference recipe:
Thanks to Jackrong for the original Qwopus3.6 model, and to groxaxo for GPTQ-Pro and the Qwen3.6 GPTQ-Pro recipe this quantization was aligned with.
Quantization recipe
| Setting | Value |
|---|---|
| Method | GPTQ-Pro / GPTQModel |
| Bits | 4 |
| Group size | 128 |
| Symmetric quantization | true |
| Desc act | false |
| True sequential | true |
| Calibration dataset | WikiText-2 raw train |
| Calibration samples | 256 |
| Sequence length | 2048 |
| MSE | 2.0 |
| Damp percent | 0.05 |
| Damp auto increment | 0.01 |
| FOEM alpha | 0.25 |
| FOEM beta | 0.2 |
| Batch size | 1 |
Preserved modules include vision, lm_head, embeddings, and norms.
Validation showed that this artifact preserves MTP-related configuration metadata, but does not include actual mtp.* tensors in model.safetensors.index.json, so this release should be treated as non-MTP for vLLM speculative decoding.
Post-save compatibility patch:
pad_token_id=248055- tokenizer class patched to
Qwen2TokenizerFastwhen needed for vLLM compatibility
Intended serving setup
This checkpoint is intended for text-only vLLM serving on RTX 3090-class hardware.
Recommended vLLM options:
vllm serve XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-FOEM-4bit-g128-ns256-v2 \
--served-model-name qwopus3.6-27b-v2-gptq-pro-foem-4bit-g128-ns256-v2 \
--language-model-only \
--dtype float16 \
--quantization gptq_marlin \
--disable-custom-all-reduce \
--tensor-parallel-size 1 \
--max-model-len 131072 \
--max-num-seqs 1 \
--kv-cache-dtype fp8_e5m2 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--enable-prefix-caching \
--max-cudagraph-capture-size 32 \
--gpu-memory-utilization 0.95 \
--trust-remote-code
Serving context for the published vLLM measurements:
The vLLM numbers were collected on an internal llm-residency deployment. The custom image recipe is not published yet, so this card does not present that image as a public reproduction target. The stable serving knobs captured from the run are listed for context.
| Field | Value |
|---|---|
| Nomad job profile | vllm-qwopus36-base |
| Served model name | qwopus3.6-27b-v2-gptq-pro-foem-4bit-g128-ns256-v2 |
| Critical flags | --dtype float16, --quantization gptq_marlin, --kv-cache-dtype fp8_e5m2, --reasoning-parser qwen3, --tool-call-parser qwen3_coder, --max-model-len 131072; --max-num-batched-tokens was not set explicitly. |
Public vLLM Reproducibility
This artifact has a public reproducibility path on the unmodified upstream vLLM OpenAI image:
- image:
docker.io/vllm/vllm-openai:nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8 - vLLM version observed in validation:
0.20.1rc1.dev16+g7a1eb8ac2 - GPU class: single RTX 3090 24 GB / Ampere
--enforce-eagerwas not used- no local sleep/wake patch or
localhost/*sleepwake*image is required for the validation below
Validated serving shape:
- context:
--max-model-len 131072 --language-model-only,--dtype float16,--quantization gptq_marlin--kv-cache-dtype fp8_e5m2,--enable-prefix-caching,--max-num-seqs 1--max-cudagraph-capture-size 32,--gpu-memory-utilization 0.95--reasoning-parser qwen3,--tool-call-parser qwen3_coder--enable-sleep-modewas included in the validation command
Startup note: this dense 27B profile can fail the first cold start after torch compile/profiling with a pessimistic KV-cache check. The public validation passed on the second start when reusing persistent vLLM/Nomad-style cache directories such as TORCHINDUCTOR_CACHE_DIR=/data/vllm-qwopus36-base/torch_compile_cache and VLLM_CACHE_ROOT=/data/vllm-qwopus36-base. Treat startup retry plus persistent compile cache as part of the serving recipe.
Reasoning / thinking mode
This model preserves Qwen3-style reasoning behavior. The validation workload below was run with thinking enabled.
MTP / speculative decoding status
This ns256-v2 artifact should be considered text-only and non-MTP for vLLM speculative decoding as published. config.json advertises mtp_num_hidden_layers=1, but the weight index does not contain source mtp.* tensors. Enabling vLLM MTP against this unpatched artifact produced essentially zero accepted draft tokens and poor throughput.
A separate experimental follow-up artifact restores real MTP tensors and quantizes the large MTP linears:
XReyRobert/Qwopus3.6-27B-v2-MTP-GPTQ-Pro-v1
That MTP-GPTQ artifact works and reaches good draft acceptance, but it was still slower than this non-MTP baseline on a single RTX 3090. For practical 100k-131k serving on 1x RTX 3090, this ns256-v2 non-MTP artifact remains the preferred choice.
RTX 3090 validation status
This checkpoint was validated on an RTX 3090 24GB with vLLM, max_model_len=131072, kv_cache_dtype=fp8_e5m2, prefix caching enabled, and thinking enabled.
Observed vLLM multi-turn agent workload metrics:
| Metric | Observed value | Notes |
|---|---|---|
| Requests observed | 15 | Multi-turn agent session calls |
| vLLM request success count | 15/15 | No vLLM errors observed during the sample |
| Average prompt size | 33,172 tokens | Real multi-turn workload |
| Average output size | 322 tokens | Real generated responses |
| Average time to first token | 5.70s | Prometheus TTFT summary |
| Average end-to-end request latency | 13.07s | Includes prefill, decode, and serving overhead |
| Average time per output token | 0.0230s/token | vLLM TPOT summary |
| Decode throughput from TPOT | about 43.5 tok/s | Decode-only estimate |
| Prefix cache hit ratio | 83.2% cumulative | vLLM prefix-cache counters |
| Live 60s prompt throughput | about 1,917 prompt tok/s | Aggregate observed window |
| Live 60s generation throughput | about 19.1 generated tok/s | Aggregate over full window, including prefill and idle mix |
| Live 60s prefix-cache hit ratio | 78.9% | Delta over the observed window |
These are practical multi-turn serving metrics, not a synthetic benchmark. They are useful for RTX 3090-class long-context serving expectations, especially multi-turn usage with prefix caching.
📊 4. Evaluation & Benchmarks
<div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; border: 1px solid #cbd5e1; border-radius: 16px; box-shadow: 0 10px 15px -3px rgba(0, 0, 0, 0.05), 0 4px 6px -2px rgba(0, 0, 0, 0.05); overflow: hidden; background: #ffffff; margin-bottom: 30px;"> <div style="background: linear-gradient(135deg, #7c3aed 0%, #4f46e5 100%); padding: 20px; color: white;"> <h3 style="margin: 0; font-size: 20px; font-weight: 700; display: flex; align-items: center; gap: 8px; color: white; border: none;">📊 Evaluation & Performance Metrics</h3> <p style="margin: 4px 0 0 0; font-size: 13px; color: #ddd6fe;">24 May update: MMLU-Pro selected subset quality check for the GPTQ-Pro ns256 quantization, plus RTX 3090 vLLM serving metrics.</p> </div> <div style="padding: 24px; display: flex; flex-direction: column; gap: 24px;"> <div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); gap: 16px;"> <div style="border: 1px solid #e2e8f0; padding: 16px; border-radius: 10px; background: #f8fafc; text-align: center; box-shadow: inset 0 2px 4px rgba(0,0,0,0.02);"> <span style="font-size: 11px; font-weight: 700; color: #7c3aed; text-transform: uppercase; display: block; margin-bottom: 6px; letter-spacing: 0.5px;">📚 MMLU-Pro Subset</span> <span style="font-size: 24px; font-weight: 800; color: #1e293b; display: block;">90.57%</span> <span style="font-size: 11px; color: #64748b; font-weight: 500;">317 / 350 resolved local run</span> </div> <div style="border: 1px solid #e2e8f0; padding: 16px; border-radius: 10px; background: #f8fafc; text-align: center; box-shadow: inset 0 2px 4px rgba(0,0,0,0.02);"> <span style="font-size: 11px; font-weight: 700; color: #7c3aed; text-transform: uppercase; display: block; margin-bottom: 6px; letter-spacing: 0.5px;">🧠 vs Qwopus BF16</span> <span style="font-size: 24px; font-weight: 800; color: #10b981; display: block;">+3.14 pp</span> <span style="font-size: 11px; color: #64748b; font-weight: 500;">306 / 350 reference CSV</span> </div> <div style="border: 1px solid #e2e8f0; padding: 16px; border-radius: 10px; background: #f8fafc; text-align: center; box-shadow: inset 0 2px 4px rgba(0,0,0,0.02);"> <span style="font-size: 11px; font-weight: 700; color: #7c3aed; text-transform: uppercase; display: block; margin-bottom: 6px; letter-spacing: 0.5px;">⚖️ vs Qwen3.6 BF16</span> <span style="font-size: 24px; font-weight: 800; color: #10b981; display: block;">+5.71 pp</span> <span style="font-size: 11px; color: #64748b; font-weight: 500;">297 / 350 reference CSV</span> </div> <div style="border: 1px solid #e2e8f0; padding: 16px; border-radius: 10px; background: #f8fafc; text-align: center; box-shadow: inset 0 2px 4px rgba(0,0,0,0.02);"> <span style="font-size: 11px; font-weight: 700; color: #7c3aed; text-transform: uppercase; display: block; margin-bottom: 6px; letter-spacing: 0.5px;">⚡ RTX 3090 vLLM</span> <span style="font-size: 24px; font-weight: 800; color: #1e293b; display: block;">43.48</span> <span style="font-size: 11px; color: #64748b; font-weight: 500;">completion tok/s, request wall-time</span> </div> </div> <div style="border: 1px solid #cbd5e1; border-radius: 12px; overflow: hidden; box-shadow: 0 2px 4px rgba(0,0,0,0.02);"> <div style="background: linear-gradient(135deg, #f8fafc 0%, #f1f5f9 100%); padding: 12px 16px; border-bottom: 1px solid #cbd5e1; font-weight: 700; font-size: 14px; color: #1e293b; display: flex; align-items: center; gap: 8px;"> <span>📚</span> 4.1 MMLU-Pro Selected Subset - 24 May update </div> <div style="padding: 16px; display: flex; flex-direction: column; gap: 16px;"> <div style="background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 8px; padding: 12px 14px; font-size: 13px; color: #334155; line-height: 1.6;"> <b>Evaluation format:</b> This uses the same 350-question MMLU-Pro subset published in the <code>test_data</code> directory of <code>Jackrong/Qwopus3.6-27B-v2</code>: 7 categories, 50 questions per category. This is not a full MMLU-Pro leaderboard run. </div> <div style="background: #eff6ff; border: 1px solid #bfdbfe; border-radius: 8px; padding: 12px 14px; font-size: 13px; color: #1e3a8a; line-height: 1.6;"> <b>Protocol note:</b> The primary <b>317 / 350 = 90.57%</b> score is rescored with the official MMLU-Pro <code>compute_accuracy.py</code> L2 answer extractor, without random fallback. The generation prompt follows the MMLU-Pro <code>run_gpt4o.py</code> OpenAI-compatible style (<code>Options are:</code>, <code>(A):</code>, and the system instruction asking for <code>The answer is ...</code>); it is not borrowed from the Qwopus model card or test set. The current MMLU-Pro <code>evaluate_from_apiX.py</code> extractor, which selects the last matching answer, scores the same outputs at <b>320 / 350 = 91.43%</b>. This card reports the conservative L2/no-random-fallback score as the headline. </div> <div style="overflow-x: auto; width: 100%;"> <table style="display: table; width: 100%; min-width: 100%; table-layout: fixed; border-collapse: collapse; font-size: 13px;"> <thead> <tr style="background: rgba(124, 58, 237, 0.05);"> <th style="width: 34%; padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: left; color: #7c3aed; font-weight: bold;">Model / run</th> <th style="width: 18%; padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: right; color: #7c3aed; font-weight: bold;">Correct / Total</th> <th style="width: 16%; padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: right; color: #7c3aed; font-weight: bold;">Accuracy</th> <th style="width: 16%; padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: right; color: #7c3aed; font-weight: bold;">Δ vs Qwen</th> <th style="width: 16%; padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: right; color: #7c3aed; font-weight: bold;">Δ vs Qwopus</th> </tr> </thead> <tbody> <tr style="background: rgba(16, 185, 129, 0.06);"> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); font-weight: 700; color: #047857;">This GPTQ-Pro ns256 artifact</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: 700; color: #047857;">317 / 350</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: 800; color: #10b981; font-size: 14px;">90.57%</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: 700; color: #10b981;">+5.71 pp</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: 700; color: #10b981;">+3.14 pp</td> </tr> <tr> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); font-weight: 600;">Qwopus3.6-27B-v2 reference CSV</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right;">306 / 350</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: bold;">87.43%</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; color: #10b981; font-weight: 700;">+2.57 pp</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; color: #64748b;">baseline</td> </tr> <tr> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); font-weight: 600;">Qwen3.6-27B-v2 reference CSV</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right;">297 / 350</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: bold;">84.86%</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; color: #64748b;">baseline</td> <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; color: #dc2626;">-2.57 pp</td> </tr> </tbody> </table> </div> <div style="overflow-x: auto; width: 100%;"> <table style="display: table; width: 100%; min-width: 100%; table-layout: fixed; border-collapse: collapse; font-size: 12px;"> <thead> <tr style="background: rgba(124, 58, 237, 0.03);"> <th style="width: 28%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: left;">Category</th> <th style="width: 22%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right;">Qwen3.6-27B</th> <th style="width: 25%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; color: #7c3aed;">Qwopus3.6-27B-v2</th> <th style="width: 25%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; color: #047857;">This GPTQ-Pro ns256</th> </tr> </thead> <tbody> <tr><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; font-weight: 600;">Biology</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right;">96%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #7c3aed;">96%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #047857;">94%</td></tr> <tr><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; font-weight: 600;">Business</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right;">88%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #7c3aed;">94%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #10b981;">96%</td></tr> <tr><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; font-weight: 600;">Computer Science</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right;">82%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #7c3aed;">84%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #047857;">82%</td></tr> <tr><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; font-weight: 600;">Mathematics</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right;">90%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #7c3aed;">88%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #10b981;">98%</td></tr> <tr><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; font-weight: 600;">Physics</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right;">76%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #7c3aed;">86%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #10b981;">92%</td></tr> <tr><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; font-weight: 600;">Chemistry</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right;">74%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #7c3aed;">80%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #10b981;">86%</td></tr> <tr><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; font-weight: 600;">Health</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right;">88%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #7c3aed;">84%</td><td style="padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; font-weight: 700; color: #10b981;">86%</td></tr> </tbody> </table> </div> <p style="margin: 0; font-size: 13px; color: #475569; line-height: 1.6;"> <b>Summary:</b> On the selected 350-question MMLU-Pro evaluation set, this GPTQ-Pro ns256 artifact reached <b>90.57%</b> accuracy using the official MMLU-Pro L2 answer extractor without random fallback. This is above the published Qwopus3.6-27B-v2 reference CSV at <b>87.43%</b> and Qwen3.6-27B-v2 at <b>84.86%</b>, but the comparison should be treated cautiously because the local result is a resolved mixed run rather than a single-pass uniform evaluation, and it uses the MMLU-Pro <code>run_gpt4o.py</code> prompt style rather than the current local/vLLM prompt template. </p> <div style="display: inline-flex; align-items: center; width: fit-content; background: #ffffff; border: 1px solid #e2e8f0; border-radius: 999px; padding: 6px 10px; font-size: 11px; color: #64748b; font-weight: 600; box-shadow: 0 1px 2px rgba(15,23,42,0.04);"> Scope note: non-truncated answers were kept, and only length-truncated cases were regenerated with larger output caps, up to max_tokens=16384. Prompts and instructions were not changed. Three pathological cases still ended with finish_reason=length and no parsed answer. For leaderboard-style comparability, the preferred next run is a single-pass full MMLU-Pro evaluation generated from the official GitHub scripts and submitted in the JSON/CSV format expected by the MMLU-Pro HF Space. </div> </div> </div> <div style="border: 1px solid #cbd5e1; border-radius: 12px; overflow: hidden; box-shadow: 0 2px 4px rgba(0,0,0,0.02);"> <div style="background: linear-gradient(135deg, #f8fafc 0%, #f1f5f9 100%); padding: 12px 16px; border-bottom: 1px solid #cbd5e1; font-weight: 700; font-size: 14px; color: #1e293b; display: flex; align-items: center; gap: 8px;"> <span>⚡</span> 4.2 RTX 3090 vLLM Runtime Notes </div> <div style="padding: 16px;"> <div style="overflow-x: auto;"> <table style="width: 100%; border-collapse: collapse; font-size: 13px; min-width: 500px;"> <thead> <tr style="background: rgba(124, 58, 237, 0.05);"> <th style="padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: left; color: #7c3aed; font-weight: bold;">Metric</th> <th style="padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: right; color: #7c3aed; font-weight: bold;">Observed value</th> </tr> </thead> <tbody> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); font-weight: 600;">Prompt tokens</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right;">509,541</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); font-weight: 600;">Completion tokens</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right;">515,679</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); font-weight: 600;">Request elapsed sum</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right;">11,859.8s</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); font-weight: 600;">Completion throughput</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: 700; color: #7c3aed;">43.48 tok/s</td></tr> </tbody> </table> </div> </div> </div> </div> </div>Paired comparison on the same 350 question IDs is positive but should be treated cautiously because the subset is small and the local result is not a single-pass uniform run. Against Qwopus3.6-27B-v2, this run has 20 local-only correct answers and 9 Qwopus-only correct answers (+3.14 pp, McNemar p≈0.061). Against Qwen3.6-27B-v2, it has 32 local-only correct answers and 12 Qwen-only correct answers (+5.71 pp, McNemar p≈0.0037).
Compatibility notes
This artifact was built and validated for text-only vLLM serving without speculative decoding. Do not enable MTP on this artifact as published; the mtp.* tensors are absent from the weight index. Vision-related modules were not validated for vision use in this release.
Limitations
- Experimental quantization.
- MTP/speculative decoding is not supported by this published artifact because
mtp.*tensors are missing. - Quality has been checked on Jackrong's 350-question MMLU-Pro subset only; this is not a full MMLU-Pro evaluation or an official leaderboard submission.
- The subset result uses the official MMLU-Pro answer extractor, but the prompt style is the MMLU-Pro
run_gpt4o.pyOpenAI-compatible template, not the current local/vLLM template. - RTX 3090 metrics above are observed workload numbers, not a controlled benchmark suite.
- Long-context and tool-calling workflows were validated on the described local vLLM/Hermes setup; behavior may vary on other serving stacks, hardware, or generation settings.
References
- Source model: Jackrong/Qwopus3.6-27B-v2
- GPTQ-Pro tooling: groxaxo/GPTQ-Pro
- Reference GPTQ-Pro recipe: groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit
- MMLU-Pro benchmark repository: TIGER-AI-Lab/MMLU-Pro
- MMLU-Pro HF Space / leaderboard: TIGER-Lab/MMLU-Pro
Individual project notice
This repository is an individual research project. It is not affiliated with, sponsored by, or endorsed by any employer or organization.
- Task
- Text Generation
- Library
- transformers
- Type
- qwen3_5
- License
- apache-2.0
- Created
- May 23, 2026
- Updated
- Aug 27, 2026