Qwen3.8-27B-Quark-AWQ-MXFP4-DFlash2-Paiton-RDNA4

Qwen3.8 MXFP4 + DFlash2 on regular vLLM

Current release

5 October 2026 image, with the launcher of 6 October: ghcr.io/eliovp/paiton-vllm-plugin:qwen38-rocm10-vllm029-20261005-r1@sha256:245c71d54f046f89b74dbfbd4c894003754561e55c9d215a8d3ce65364fb0f80. One image serves the MXFP4 path and the optional 3-bit weights, with prefix caching, images and up to 262,144 tokens per request. The plugin README has the downloads, the commands and the measurements; the launcher pins the image by digest, no registry login needed.

What you get on one R9700 (all commands start with bash models/Qwen3.8-MXFP4-DFlash2/):

In 20-turn coding sessions growing from about 100K to 181K tokens, the 3-bit coding mode solved 87.5 % of the tasks with no decline over the turns (MXFP4 --mode long: 83.75 %) (long-context results).

The sections below document the 20 September images and the historical v1.0.0 files of this repository. They stay runnable, but the release above replaces them.

Run the 20 September images

Use Linux x86-64, Python 3, Docker, and one Radeon AI PRO R9700 with 32 GB VRAM and working AMD GPU device access. Run the following commands from a checkout of the public plugin repository:

git clone https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin

The images contain the runtime; download the target and draft weights separately using the Hugging Face CLI (hf), or point the variables at existing copies of these exact snapshots.

The main setup below uses the updated 65K image with APC off, matching the benchmark configuration. The optional long-context profile remains available below.

This release loads the Unsloth NVFP4 checkpoint through the MXFP4 runtime path. The AMD checkpoint and automatic downloader in the historical release below are for the older images.

export PAITON_TARGET_DIR="$PWD/model-cache/qwen38-nvfp4"
export PAITON_DRAFT_DIR="$PWD/model-cache/qwen38-dflash2"
export PAITON_CACHE_DIR="$PWD/runtime-cache/qwen38-rocm10-65k-20260920"
mkdir -p "$PAITON_TARGET_DIR" "$PAITON_DRAFT_DIR" "$PAITON_CACHE_DIR"

hf download unsloth/Qwen3.8-27B-NVFP4 \
  --revision f0b7c9e722f5565102fff8481c99e4d86ae099c7 \
  --local-dir "$PAITON_TARGET_DIR"
hf download tcclaviger/Qwen3.8-27B-DFlash2-FP8 \
  --revision ee0cb26a8279b7910cc28d82a8a3e15e4728d56f \
  --local-dir "$PAITON_DRAFT_DIR"

bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10-65k.sh

The server runs in the foreground at http://127.0.0.1:18982/v1, with API model name Qwen3.8. First startup loads/converts weights and compiles runtime components; wait for readiness before sending requests or measuring throughput. The persistent cache is reused on subsequent starts. The first image pull is approximately 9.6 GB, excluding model weights.

From another terminal:

curl --fail http://127.0.0.1:18982/health
curl --fail http://127.0.0.1:18982/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.8","messages":[{"role":"user","content":"Write a Python function that removes duplicate items while preserving order."}],"temperature":0.7,"top_p":0.95,"max_tokens":256,"stream":true,"chat_template_kwargs":{"enable_thinking":false}}'

For the separately validated 200K chat configuration, stop the 65K container (docker stop paiton-qwen38-65k), keep the same target and draft directories, and select the existing long-context image:

export PAITON_CACHE_DIR="$PWD/runtime-cache/qwen38-rocm10-200k"
mkdir -p "$PAITON_CACHE_DIR"
bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10-200k.sh --profile chat --context 200000

Select one profile at a time on the GPU. The 200K image passed a 198,989-token prompt followed by generation, a fresh short request, and both ordinary and streaming XML tool calls. The performance tables below are for the 65K profile; they are not measurements of 200K throughput.

No private compiler checkout is needed to run these images. The full qualified runtime payload is distributed in the images; the public repository alone is not a complete build context for this release.

GPU, context, and memory controls

The 65K and 200K names select startup presets. Context is not compiled into the image: the launchers can set both the target and DFlash draft limits at startup, using the existing images. The limit includes input and generated tokens. A larger limit still requires sufficient cache and VRAM; it does not guarantee useful model quality at that length. The checkpoint's configured ceiling is 262,144 tokens; the largest serving limit tested here is 220,000, not a claim that 262K fits this GPU.

The launcher exposes /dev/kfd and all of /dev/dri, adds the video group, and uses host IPC. Select the GPU with your usual ROCm environment variables. For example, if the R9700 is ROCm GPU 1:

export ROCR_VISIBLE_DEVICES=1
bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10-200k.sh --profile chat

Use the index or GPU UUID appropriate to your system. --list-gpus lists physical render devices and PCI addresses for identification; render-device numbers are not ROCm visibility indices. The launcher does not choose a GPU automatically.

The launcher forwards your ROCR_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES values unchanged. If a variable is unset on the host, it is also unset in the container, overriding the image's default. The launcher does not force a GPU index or UUID. An explicitly empty mask remains empty; it does not mean "show all GPUs".

On Linux, ROCR_VISIBLE_DEVICES also filters ROCr tools such as rocminfo; HIP_VISIBLE_DEVICES applies at the HIP layer. If you set both, HIP indices refer to the GPUs remaining after the ROCr filter. Check any existing exports before launching: setting both variables to 1 does not necessarily select the second physical card. Leave unused variables unset rather than assigning empty strings. These controls do not make unsupported GPU architectures compatible with this image.

For a GPU shared with a desktop, start with the smaller preset:

bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10-65k.sh \
  --profile desktop

This selects 32,768 tokens, one scheduled request, 1,024-token prefill chunks, smaller graph captures, and a 2 GiB KV allocation. It is a starting point for sharing VRAM, not a guarantee against memory exhaustion. The unchanged benchmark presets reserve a fixed KV pool and target a dedicated GPU.

Customize the limits without rebuilding or downloading another image:

bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10-65k.sh \
  --profile desktop --context 16384

Setting memory utilization switches to automatic KV sizing unless you explicitly provide --kv-cache-memory-bytes. Lowering context alone does not reduce a fixed KV allocation. --kv-cache-memory-bytes auto also enables automatic sizing. Automatic profiling can leave insufficient cache for the requested context in this runtime; startup reports the required and available cache sizes. Use --dry-run to inspect the complete Docker command, or --help for all options. Customized settings are separate from the benchmark configuration below.

200K and 220K with prefix caching

With your GPU visibility configured as above, use the chat profile for 200K, or set a larger context on the same image:

# 200K total context, including generated tokens
bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10-200k.sh \
  --profile chat

# Stop the existing server before selecting 220K instead
docker stop paiton-qwen38-200k
bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10-200k.sh \
  --profile chat --context 220000

The chat profile uses one scheduled request, an 8 GiB KV pool, 1,024-token prefill chunks, prefix caching, and thinking disabled. It also applies the memory-allocation setting needed by this configuration. It leaves little spare VRAM: select a dedicated R9700 rather than a card driving a busy desktop. These settings differ from the 65K throughput benchmark configuration.

On one R9700, the 220K configuration correctly retrieved a value from a 215,005-token prompt. The identical repeat reused 213,840 cached tokens and completed in 1.77 seconds versus 130.00 seconds cold; both answers were nine tokens at temperature zero. These are complete response times from one functional probe, not general latency or decode-throughput claims. Changed-prefix retrieval, 380- and 409-token answers without observed looping, a subsequent XML tool call, and a fresh short request also passed.

The unchanged release presets disable APC, so zero hits with their defaults is expected. The persistent disk cache used during startup is separate from the in-memory conversation prefix cache.

--prefix-caching on enables the experimental APC configuration and selects the compatible recurrent-state settings. This changes memory requirements; do not assume the release preset's fixed cache budget remains sufficient. Use --profile chat for the complete long-context configuration. Cache hits require an unchanged token prefix that is still resident. They reduce repeated prompt work, not the cost of generating each new token.

The chat profile reports cached tokens in usage.prompt_tokens_details.cached_tokens. Streaming clients must also request "stream_options":{"include_usage":true} to receive usage in the stream. Server-side cache counters are available at http://127.0.0.1:18982/metrics.

These checks are not a full long-conversation quality evaluation. Repetition penalties and sampling settings belong in each client's API requests. Report the prompt, settings and server logs when diagnosing loops; a sampling workaround is not a general fix.

The release preset's model template enables thinking when the client omits that setting. The chat profile and our benchmarks disable it. Use --thinking off to set the server default, or send "chat_template_kwargs":{"enable_thinking":false} in each request.

Images and vision

These presets serve text only with --language-model-only. They do not accept a llama.cpp mmproj file. A separate single-image smoke test with an 8K context limit used the checkpoint's vision weights and passed color identification and subsequent text and tool requests. That configuration needs additional VRAM and is not included in these launchers; 200K/220K multimodal use has not been validated.

20 September benchmark results

R9700, 300 W; vLLM 0.29 / ROCm 10; 65,536 context; maximum eight sequences; APC off; thinking off. Temperature 0.7, top-p 0.95, top-k 20, seed 42. Each arm uses a fresh process and the same fixed warmup.

Weighted decode: 146.85 → 154.42 tok/s (+5.16%).

Concurrency values are aggregate generated tokens per second over each complete 48-request workload.

Update p50 is the median streamed-update gap, not per-token latency or TTFT. Sampled output content and accepted-token work can differ; these are serving-throughput measurements, not identical-output timing. Nominal prefill depths correspond to median actual prompt lengths 1516.5, 5894.5, 11802, 23549.5 and 47016.5. GGZ comparison values are historical references from the previous post; GGZ was not rerun.

Both arms: 290/290 successful benchmark requests. Candidate matches all 12/12 greedy controls before timing. Each arm repeats 12/12 greedy outputs after the benchmark. No unhandled serving errors.

Decode has five scored requests per category after one warmup. Prefill has eight scored requests per depth after two warmups. These are individual complete runs; category values are not selected across repeats.

Machine-readable results.

Files in this repository

The native overlay, lock files and benchmark artifacts stored here remain the historical v1.0.0 release. Use the GHCR images and GitHub instructions above for the current runtime. This repository is not a standalone Transformers checkpoint. No proprietary compiler checkout is needed to run the images.

Third-party notices · Paiton