MiniCPM5-2B-W4A16-Paiton-RDNA4

MiniCPM5-2B W4A16 for AMD RDNA4

Small local chat and coding assistance on one Radeon AI PRO R9700, gfx1201, 32 GB VRAM, with Paiton-compiled decode kernels and an OpenAI-compatible vLLM API.

This repository distributes compiled artifacts, manifests and reproduction evidence. Model weights are downloaded directly from the pinned upstream checkpoint. It is not a standalone Transformers checkpoint. Version v1.0.0.

Run

Linux x86-64, Docker and working AMD GPU access are required. No Hugging Face token is needed for the public weights.

docker run -d --name paiton-minicpm5 \
  --device /dev/kfd --device /dev/dri --ipc=host \
  -p 8036:8036 -v paiton-minicpm5-cache:/models/cache \
  ghcr.io/eliovp/paiton-vllm-plugin:minicpm5-2b-w4a16-rdna4-v1.0.0

The API listens on 0.0.0.0:8036; the named volume preserves verified weights and runtime caches. Follow docker logs -f paiton-minicpm5 while it prepares. A completed generation, not an open port alone, establishes useful readiness.

curl --fail http://localhost:8036/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"minicpm5-2b","messages":[{"role":"user","content":"Explain binary search in two sentences."}],"max_tokens":256,"temperature":0,"stream":true}'

docker exec -it paiton-minicpm5 python3 /opt/paiton/minicpm5/chat.py

For stock reproduction, stop this server and run the same Docker command with --stock after the image name. Use --download-only to prepare weights, or --offline once cached. The immutable image reference is:

ghcr.io/eliovp/paiton-vllm-plugin@sha256:b8d6bf8e6c1934a82d8c99156061e0a44aed956ea021f73276fc07bf74c88e5e

Measured loading and generation

The paired warm-chat reduction is 33.1%, with unresolved between-process and within-process variability. Earlier isolated stock and Paiton runs were faster; this is not a universal speedup guarantee against every stock observation. Paiton adds some prefill/first-token overhead while reducing decode and complete-request latency. The shared startup import fix applies to both modes. See full measurements and limitations and raw results.

Paiton prepared-cache first useful response ranged 27.13–28.77 s. With verified cold pages in an isolated checkpoint copy and prepared runtime caches, the median was 29.81 s (n=3). With warm weights and empty runtime caches it was 46.53 s (n=3). No system-wide cache dropping was used. The 2.11 GB checkpoint download took 28.50 s plus 5.72 s verification on the qualification connection; network times vary. Base image pull was not measured because layers were cached. Container storage is approximately 11.79 GB compressed / 27.84 GB allocated root filesystem, plus weights and about 39 MB prepared runtime caches; allow 60 GB for first setup. A separate llama.cpp screen started faster but is not the qualified Paiton path.

Identity, quality and limits

• Upstream: openbmb/MiniCPM5-2B-GPTQ, revision 6c1ee6fa521aa53f47cfb32696e6d8ef5b0db805, approximately 2.517B dense parameters, Apache-2.0.
• Despite the name, stored tensors use AWQ GEMM, asymmetric INT4 G128. Paiton losslessly repacks them for decode. The kernel unpacks/dequantizes to FP16 and accumulates FP32; this is not native INT4 matrix arithmetic. Original checkpoints remain unchanged. Publisher calibration data and exact BF16 source revision are not pinned; no new quantization is performed here.
• Defaults: 8,192 total context tokens, two scheduled sequences, FP16 activations and KV, 1 GiB KV allocation, thinking disabled, no prefix caching or speculation. The upstream advertised 128K context is not qualified. Thinking-on W4, LoRA and multi-GPU operation are not qualified.
• The reproducible direct-mode set scores 14/20 for both stock and Paiton, with 20/20 exact final-text matches; BF16/FP16 scores 15/20. Arithmetic, strict filtering and some coding tasks fail. This small suite does not establish general quality parity or a strong reasoning guarantee.
• Both final-image modes pass 8/8 protocol checks, including streaming, tool exchange, 8,040-token retrieval and context overflow rejection. Generated-code tests run in a resource-bounded, network-disabled container.
• Compiled artifacts have no Torch/ATen/libc10 linkage and load without importing Torch. A clean rebuild passed numerical/repack/graph tests but differs in binary SHA; byte-identical compiler reproduction across directories is not established. This repository retains the exact serving-tested artifact.

Download artifacts separately

The Docker image already contains this overlay:

hf download EliovpAI/MiniCPM5-2B-W4A16-Paiton-RDNA4 --revision v1.0.0 \
  --include 'overlay/*' --local-dir ./paiton-minicpm5-hf

To override the artifact, add these arguments before the Docker image name:

-v "$PWD/paiton-minicpm5-hf/overlay:/models/paiton-overlay:ro" \
-e PAITON_MINICPM5_ARTIFACT=/models/paiton-overlay/minicpm5_awq_decode_gfx1201.so

Keep the .so and .json together. The integration verifies geometry, gfx1201, ABI and checksum. These artifacts require the tested ROCm 7.14 ABI. The included wheel is for inspection; repository-wide dependency defaults do not qualify standalone pip installation. Use the pinned image.

Sources and reproduction

• Community feature branch — PR/merge is pending.
• Selection and dated popularity, quantization evidence, benchmark methodology, reproduction.
• Image and component pins, artifact manifest, checksums, Hub manifest.
• Model/runtime notices, runtime dependencies, licenses.

The Studio adapter is prepared separately and uses existing model-loading, persistent caching and exclusive GPU-leasing conventions. Publishing these artifacts does not update an active Studio installation.