Downloads · 30 days
255
39% of all-time downloads
hugypufy/Swift-Qwen3.8-27B-PARO-int5
Swift-Qwen3.8-27B-PARO-int5 is a image-text-to-text model from hugypufy. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as other.
ukisai/Swift-Qwen3.8-27b with ParoQuant rotations and int5 weights, for AMD Radeon AI PRO R9700 (RDNA4) on a patched vLLM.
Downloads · 30 days
255
39% of all-time downloads
All-time downloads
658
Public
Parameters
7.1B
21.9 GB on disk
Likes
4
Public
Click a slice to open those files.
.safetensors21.8 GB · 100%
How the weights are stored.
I323.8B · 54%
From the Hugging Face model README
ukisai/Swift-Qwen3.8-27b with ParoQuant rotations and int5 weights, for AMD Radeon AI PRO R9700 (RDNA4) on a patched vLLM.
Size: 21.8 GB (bf16: 55.6 GB).
This is a 27B model; the Hub's params badge counts packed tensor elements.
| this model | bf16 | |
|---|---|---|
| WikiText-2 perplexity | 7.092 | 7.098 |
| KL divergence vs bf16 (top-256, nats) | 0.0126 | 0 |
| Top-1 token match with bf16 | 95.04% | 100% |
| MMLU 5-shot | 82.8% | — |
| GSM8K flexible / strict | 96.6% / 93.7% | — / — |
| HumanEval / HumanEval+ | 97.6% / 93.9% | — / — |
bf16 = the unquantized source model served on the same image; — = not measured.
Perplexity is cross-stack — this model at tensor parallel 1, bf16 at 2 — and wikitext-2's train split is in the stage-2 calibration mix, so a reading at or below bf16 is not evidence of a general gain.
GSM8K and HumanEval are chat-templated with reasoning on, greedy, on a Qwen chat template at reasoning_effort=xhigh rather than the default template the command below uses; both scores move with that choice.
2× Radeon AI PRO R9700, TP=2, DFlash2-FP8 drafter (7 tokens), KV pinned with KV_MEM=15000000000, tokens/s.
| Single-stream decode | 160.1 |
| Concurrent decode c1 / c2 / c4 / c8 | 134.8 / 237.4 / 344.2 / 416.8 |
| Prefill 2k / 8k / 16k / 32k / 60k | 3017 / 3214 / 3161 / 3058 / 2953 |
| KV cache tokens | 762,413 |
| Weights per GPU | 11.73 GiB |
Each row is a single measured run, on a newer radiance build than the 0.9.3 image the setup below installs, so expect these figures to move. KV capacity follows the pin, not the weight format.
Requirements
amdgpu kernel driver loaded, so /dev/kfd and /dev/dri exist. ROCm userspace ships inside the image, so there is no host ROCm install.docker or podman (auto-detected), and a host git and python3 (standard library only).stilldeadcode/vllm-radiance:0.9.3 image, both of which setup fetches.Two ways to get the code. The launcher these instructions use is open as a pull request to ggz14/radiance-vllm-mxfp4 (#46). Clone the fork below to run it today, or wait for the pull request to merge and clone ggz14's repository instead — the commands are the same either way.
Install and serve
git clone https://codeberg.org/hugypufy/radiance-vllm-mxfp4 && cd radiance-vllm-mxfp4
SRC_REPO=hugypufy/Swift-Qwen3.8-27B-PARO-int5 SNAP=~/models/Swift-Qwen3.8-27B-PARO-int5 \
./setup-paroquant.sh --int5 --yes
MODEL_DIR=Swift-Qwen3.8-27B-PARO-int5 MODE=prod SPEC=7 ./paroquant/run_paroquant.sh
Setup is one-time and safe to re-run. The launcher reads the quantization format from config.json and sets the kernel knobs it needs, so that is the whole serve command; the server answers as Qwen3.8-PARO on http://localhost:8080/v1. Every start patches vLLM inside the container and compiles the ParoQuant kernels for gfx1201, about two minutes, before the model loads; the first start also compiles Triton and inductor graphs, which adds several minutes and is cached for later starts.
Test
curl -s http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"Qwen3.8-PARO","messages":[{"role":"user","content":"Hello!"}]}'
Options — set as environment variables on the serve command.
| Variable | Default | Notes |
|---|---|---|
KV_MEM | unset; the pool is sized from GPU_UTIL (0.92) | Pin the KV cache in bytes per GPU; the figures above used KV_MEM=15000000000 |
SPEC | 5 | DFlash2 draft tokens per step. The numbers above used 7 |
CHAT_TEMPLATE | the repository's qwen-fixed-v22.3.jinja | Any .jinja on the host, for example this checkpoint's own chat_template.jinja |
RADIANCE_VERIFY_HEAD | 1 | Set to 0 to use prompt_logprobs, which hangs at the default |
linear_attn.in_proj_a/b, linear_attn.conv1d / A_log / dt_bias, lm_head, embeddings, norms, and the rotation theta / channel_scales (pairs are int16). MTP head not included.Governed by the base model's Swift Open License v1.0 (terms); quantization does not relicense the weights. Modified derivative, not made or endorsed by UkisAI. Rotations from z-lab/Qwen3.8-27B-PARO (Apache-2.0); original base Qwen/Qwen3.8-27B.