GLM-5.3-Flash-rocm-r9700

<div align="center">

GLM-5.3-Flash · ROCm/RDNA4 serving overlay & quantization recipe

Serve zai-org/GLM-5.3-Flash on 8× AMD Radeon R9700 (gfx1201 / RDNA4)

</div>

---

Overview

This repository contains the serving overlay and quantization recipe used to run a self-quantized derivative of GLM-5.3-Flash — a 321B-parameter sparse MoE with native multimodality — on a cluster of 8× AMD Radeon R9700 GPUs (gfx1201 / RDNA4).

Stock vLLM does not ship this architecture on the RDNA4 path, and the aiter package is physically absent from the base image. This repo is the bring-up work that makes it serve correctly and fast on consumer RDNA4 hardware, plus the quantization recipe that produced the published weights.

📦 Weights: GLM-5.3-Flash-RFA-RFI8-8xR9700 — 197.8 GB, 25 shards, 4.93 bpw average.

---

Table of contents

• Attribution & credits
• The model
• What this repository does
• Repository layout
• Quantization
• Build
• Run
• Performance
• Multimodal policy
• Known limitations
• License

---

Attribution & credits

This work stands on the shoulders of two upstream projects. Credit is deliberately split:

In short: the runtime and the quant kernels are tcclaviger/vllm (IronLLM Labs); our contribution is the GLM-5.3-Flash overlay + RDNA4 port + the quant recipe layered on top.

---

The model

GLM-5.3-Flash is Z.ai's first natively multimodal GLM-5 model — 321.3B total / ~18B active parameters with a hybrid attention backbone:

---

What this repository does

1. glm5next backport into tcclaviger/vllm:latest — registers Glm5NextForConditionalGeneration (upstream vLLM PR #53906 backport) and the Glm5Next processor / image / video tower in the v0.24 runtime.
2. kpool + sparse-MLA no-AITER port under RDNA4 — the DSA layers boot without aiter: indexer direct-call, compact profiler workspace, torch top-k fallback, cudagraph-safe Triton DSA kernels, and a full kpool custom op on ROCm.
3. Sync-free prepare_chunk_indices — removed the host DtoH sync in the KDA region, which unblocks FULL-cudagraph capture at batch size ≥ 2.
4. Multimodal resize clamping — images upscaled to min 384×384, downscaled to max 1280×1280.
5. RFA + RFI8 composite quantization — the recipe that produced the published weights.

---

Repository layout

.
├── app/vllm/vllm/…              # patch overlay tree (copied over the container's /app/vllm)
├── apply_overlay.sh              # patches-at-start: applies the overlay + purges stale bytecode
├── run-glm53.sh                  # canonical docker run wrapper (serving)
├── preflight-gpu.sh              # GPU/host preflight checks
├── tests/                        # kernel/validation probes (kpool, mhc, no-aiter DSA, R2b)
├── OVERLAY_NOTES.md              # technical bring-up notes (root-cause writeups)
└── quant/                        # quantization recipe + injector patches
    ├── glm5_archspec.py          # ArchSpec for glm5_next (patch + --check validator)
    ├── glm5_fp8_source_patch.py  # FP8 e4m3 block-source dequant support
    ├── glm5_kda_remap_patch.py   # KDA module-tree <-> checkpoint name bridging
    ├── run-quant.sh              # container-side wrapper for the composite quantizer
    └── run-full.sh               # full 8×R9700 quantization run (composite recipe)

---

Quantization

The checkpoint mixes three schemes in one quant_method: "rfi" composite, produced by the tcclaviger/vllm composite router (--quantize --mixedQuant, activation-aware):

Average bits-per-weight

bpw = (total safetensors bytes × 8) / total parameters
    = (197,843,715,288 × 8) / 321,342,220,638
    = 4.9254  ≈  4.93 bpw

The full recipe (archspec, source patches, kda-remap, run scripts) lives in quant/.

---

Build

The overlay is applied at container start — no image rebuild required:

# 1. pull the base image (vLLM 0.24 tree, torch 2.11 / ROCm 7.2.3, RDNA4/gfx1201)
docker pull tcclaviger/vllm:latest

# 2. the container mounts this repo at /overlay and runs apply_overlay.sh first
#    (see run-glm53.sh for the exact docker run invocation)

---

Run

# current production configuration (see run-glm53.sh for the full wrapper)
docker run --rm --tty --ipc=host --shm-size=128g \
  --device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \
  -v /path/to/GLM-5.3-Flash-RFA-RFI8-draftRFA4:/models:ro \
  -v "$PWD":/overlay:ro \
  --entrypoint bash tcclaviger/vllm:latest \
  -c "/overlay/apply_overlay.sh && GLM5_NEXT_MTP_PROPOSER=1 exec vllm serve /models/GLM-5.3-Flash-RFA-RFI8-draftRFA4 \
      --served-model-name glm53-flash --trust-remote-code --quantization rfi \
      --tensor-parallel-size 8 --gpu-memory-utilization 0.9575 \
      --max-model-len 300000 --max-num-seqs 4 --max-num-batched-tokens 2048 \
      --kv-cache-dtype fp8 \
      --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' \
      --enable-prefix-caching --distributed-executor-backend mp \
      --compilation-config '{\"cudagraph_capture_sizes\":[1,2,4,8,16],\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_copy_inputs\":true}'"

run-glm53.sh is the full wrapper with the RDNA4-specific environment (HIP/RCCL, tunableop, RPC timeouts, the no-AITER rung ladder) and the production cudagraph config. The production launch exposes the MTP proposer via the GLM5_NEXT_MTP_PROPOSER=1 gate — without it the stock EagleProposer path is used (see OVERLAY_NOTES.md).

Note on the checkpoint name: -draftRFA4 is a 183 GB requant variant of the published weights in which the nextn/MTP drafter block is compressed to 4-bit RFA — it frees VRAM so the MTP drafter + fp8 KV cache fit alongside the 300k context window. The published full checkpoint (197.8 GB, BF16 drafter) serves the same stack with MTP disabled.

---

Performance

Verified numbers on 8× R9700 (256 GB VRAM, TP8), current production config (MTP spec=3, fp8 KV, FULL_AND_PIECEWISE cudagraphs, VLLM_USE_NCCL_SYMM_MEM=0):

MTP speculative decoding metrics (live engine log, spec=3): mean acceptance length ~3.7–3.9 of 4 draft tokens, per-position acceptance ~1.00 / 0.95–0.99 / 0.85–0.92, average draft acceptance 91–97%. The VLLM_USE_NCCL_SYMM_MEM=0 + CLAV_AR_QUANT_MIN_KB=128 allreduce combo was A/B-verified: +9.9% on bs=1 decode, neutral at bs=4, quality PASS.

---

Multimodal policy

Images are resized with aspect ratio preserved, clamped to min 384×384 and max 1280×1280. The processor is invoked with a min/max image-token budget (--mm-processor-kwargs).

---

Known limitations

• MTP is OFF RESOLVED (2026-09-09). MTP spec-decode now ships enabled in production (spec=3, up to ~88 tok/s single-request). The fix is a dedicated glm5next_mtp.py proposer with per-group draft KV plumbing (own block table + slot mapping per draft KV cache group), enabled via the GLM5_NEXT_MTP_PROPOSER=1 gate; the stock EagleProposer still fails on the "All drafting layers should belong to the same kv cache group" assertion. Details in OVERLAY_NOTES.md.
• fp8 KV: works in production (--kv-cache-dtype fp8, scales fixed at 1.0, no runtime calc_kv_scales). Do not enable --calculate-kv-scales — runtime calibration happens on the profile dummy-run where the KDA recurrent state is uninitialized, producing garbage scales and hard output looping. See OVERLAY_NOTES.md.
• Chat needs reasoning_effort="low". The GLM-5.3 chat template defaults to Reasoning Effort Max, which spends 16k+ tokens thinking before producing content on long generations.

---

License

MIT — see LICENSE. The GLM-5.3-Flash base model is © Z.ai (zai-org), MIT license. The serving stack tcclaviger/vllm:latest is © IronLLM Labs (see its Docker Hub page).