GLM-5.3-Flash-NVFP4-h

GLM-5.3-Flash-NVFP4-h — W4A16 dense-side quantization

English · 日本語 · 简体中文 · 한국어

Current tag: v1 — the only checkpoint published so far; a v2 checkpoint would have to clear the gate described in Versions. The serving scripts received a minor update on 2026-09-19 (scripts only — see Serving script additions).

A derivative of the official nvidia/GLM-5.3-Flash-NVFP4 checkpoint in which the layers NVIDIA's release left in BF16 — the attention linears (KDA fused in_proj/out projections, MLA q/kv/o, indexer wq_b), the shared experts (gate/up/down), and lm_head — are quantized to W4A16 NVFP4 (weight-only 4-bit, group size 16). Routers, norms, embeddings, the vision tower, and the MTP layer (layers.45) stay BF16; the routed experts keep the stock W4A4 NVFP4 quantization. enjoying the incomplete

Read this before you download

This checkpoint does not boot on the stock vLLM image. It needs the kda-quant and mla-quant source overlays from the companion repository (plus glm5next-mtp-bf16 when you serve with speculation), bind-mounted into the serving container as documented there. Served as a drop-in nvidia/GLM-5.3-Flash-NVFP4 replacement it fails to load.

Those overlays are my own local patches for my own two-node machine. I have not proposed them upstream and they are not on anyone's upstream roadmap, so please treat them as something you patch in yourself, not as a feature arriving in a future vLLM release.

The configured ceiling is 204,800 tokens, generation included — that is a limit on the whole sequence, not on the input alone with room to generate on top of it. My recommended configuration is EP off. On the shipped EP-off configuration, the longest input I have confirmed working is 197,485 tokens, which is a separate fact from the 204,800 ceiling above — I have not yet cleared the full width of that window or the stability of running long contexts concurrently, and I am not adding a concurrent long-context guarantee.

If you serve this through pi, pi saves conversation history automatically, sets contextWindow to 163,840, and with its default settings starts auto-compaction around 147,456. That 147,456 figure is pi's own recommended client-side setting; it does not mean the model's context capacity has been changed to 147,456 — keep those two facts separate.

--max-num-seqs 20 is a scheduler-side upper bound on admitted requests, not a guarantee that 20 of them run concurrently; how many actually run together is decided by the KV pool (see the speed table below).

Reaching READY does not mean host memory is done growing — more of it is claimed later, the first time a large prefill shape arrives, and a rare engine stop is possible. If the engine stops, I recover it and resume only the unfinished work from the saved session and output files; a generation that was in progress can be lost. The two-node boot check below (serve/check-headroom.sh) is an optional diagnostic I run by hand when investigating a boot, not a required gate before you send traffic — passing it is not a guarantee against a later failure, and a low reading by itself does not refuse the boot.

Purpose: single-stream decode speed on 2 NVIDIA DGX Spark nodes ( TP=2 over Ray, NCCL over RDMA). On my fixed ruler — 64 Japanese prose prompts, max_tokens=512, temperature=0, thinking off, C=1 — this checkpoint plus the model's own MTP draft at K=2 reaches 35.09 tok/s on node pair 1, expert parallel on, NCCL over RDMA. The same checkpoint, pair, K and EP over TCP sockets reads 24.01 tok/s on the same ruler. Every other number, with its pair / transport / prompt count / K / EP, is in the table below; I do not quote a ratio between rows whose conditions differ.

Several improvement ideas are still on my list, but the model has reached a level I find usable, so I am releasing it now. It is a work in progress: please take it and move it toward your own idea of complete. I will keep working and publish updates as results come in, and I would be glad to see your versions on Hugging Face and GitHub too.

Base checkpoint

• nvidia/GLM-5.3-Flash-NVFP4 at revision 09b04e5e74bca08ca8549fc736d4cdd8624bfde3 (pinned).
• Upstream base model: zai-org/GLM-5.3-Flash (MIT).

How it was made

Produced by requant/requant.py --target h in the companion repository: https://github.com/tenhkspark/glm53-flash-nvfp4-2node. The script reads the official checkpoint read-only, rewrites the target tensors into fresh safetensors shards, and rewrites config.json / hf_quant_config.json / model.safetensors.index.json for the modelopt MIXED_PRECISION loader. Non-target tensors are copied through unchanged.

The MTP draft checkpoint

The --speculative-config in the serving command below points at /checkpoint-mtp, a separate BF16 draft directory. It is not part of this repository — you build it, and the companion repository builds it for you in STEP 8 (build-mtp-draft) of scripts/agent-run.sh, which runs:

python3 requant/build-mtp-draft.py \
  --target /path/to/GLM-5.3-Flash-NVFP4 \
  --out /path/to/checkpoint-mtp

The MTP layer is layers.45, which the NVIDIA release already stores in BF16 and which this repository carries through unchanged. The script copies those tensors as-is, adds the two keys the vLLM draft does not share with the target (embed_tokens and shared_head.head, sourced from embed_tokens and lm_head), and writes a flattened text-only config.json with the quantization section dropped. Point --target at the stock nvidia/GLM-5.3-Flash-NVFP4 directory rather than at this one: route h quantizes lm_head to NVFP4 and the draft has to be BF16. The output is roughly 16 GiB and each node needs its own copy.

Quality delta (gate v2, measured)

I evaluated this checkpoint against a stock baseline capture using the same serving stack on 2026-09-16:

Perplexity. I measure it on eight short Japanese sentences (249 characters in total) written for this test and defined inline in requant/verify.py; they are not drawn from any public corpus and I claim no training-set exclusion. Each sentence goes to the served endpoint as a prefill-only request (/v1/completions with echo=true, max_tokens=1, prompt_logprobs=1, temperature 0); the log-probabilities of the actual prompt tokens are accumulated across all eight sentences, and the mean negative log-likelihood over that single pooled token stream is exponentiated into one number. Stock and candidate are measured by the identical function against the same served model: route h scored 12.543 against 11.938 for stock, a ratio of 1.051. The sample is small, so this checks for gross degradation, not for parity. The TTFT gate reuses the same eight sentences, repeated twelve times, as its probe input.

eval-200. A fixed set of 200 Japanese single-turn prompts shipped in the companion repository (bench/eval-200.jsonl): 50 each of reason, trap, tool and longread, mixed easy/medium/hard. Every item is answered greedily on the served endpoint (temperature 0, max_tokens 384, thinking skipped via an empty assistant continuation) and graded by a deterministic rule in requant/verify.py: whitespace-insensitive exact match when the item's rubric demands the answer alone, substring containment otherwise; trap items must carry a refusal and no digits, tool items must name the required functions in order plus the expected final value. The tool column is 0/50 on every checkpoint by construction -- the prompts never name a callable function and no tool schema is sent -- so the gate scores the remaining 150 live items and requires the candidate to stay within 0.02 of stock with zero request errors (0.447 stock -> 0.453 route h). The raw totals 67/200 -> 68/200 are reported for transparency only; they include that permanent zero column and are not the pass/fail criterion.

Accuracy by category (50 items each):

The category moves are not independent: the trap grader rewards refusal, so a more hedging model scores higher there and lower on reason. A net +1 total can be produced by a directional degradation.

The perplexity ratio of 1.051 is a 5.1% increase over the stock model: inside the declared gate threshold, but an increase rather than strict parity. Eval-200 moved by +0.0067 on the 150 live items the gate scores. A 150-item set resolves accuracy to roughly +-0.08 at 95% confidence (paired, assuming ~20% of items flip), so the +0.0067 I measured is indistinguishable from zero — and so would be a true regression of 0.05. The 0.02 gate threshold is finer than what this set can resolve: the +-0.08 half-width is 4x the threshold. Treating that half-width as a normal sampling error (sigma = 0.08 / 1.96, about 0.041) and the gate's accuracy condition as 'observed difference at least -0.02', a checkpoint genuinely 0.06 worse than stock still clears that condition about 16% of the time, and one genuinely 0.08 worse about 7%. Those 0.06 and 0.08 are percentage-point differences in accuracy, not relative drops, and the two probabilities are for the accuracy condition alone, not for the composite gate. Reading this gate as 'no regression' is wrong; it only rules out a large one.

Every gate and eval-200 row above is Japanese, single-turn and thinking off. The one evaluation that would cover long context, parallel requests and agentic tool use — the 13-item end-to-end serve suite — did run, on 2026-09-16, on both node pairs, but 4 of its 13 items returned nothing in either run: agentic tool success, agentic final answer, integrated task en 108K and integrated task ja 108K. So agentic tool use and 108K long context are still unmeasured on this checkpoint — not because the suite is waiting to run, but because the items that cover them produced no result when it did. The two runs are also not a like-for-like route h vs stock comparison: pair 1 ran with no speculation and pair 2 ran with speculative decoding at 78.0% acceptance, so even the items that did return cannot be read as a difference between the two checkpoints.

What this gate does not measure. Every number above is Japanese, single-turn, thinking off, greedy, under 2k tokens of context and one request at a time. I have no measurement of this checkpoint on English or any other language, on code correctness, on multi-turn conversations, on tool calling (the tool column is a structural zero), on instruction following, on long-context quality — the needle probe in "Long context, measured" scores 40/40 up to 194,544 tokens, but that is retrieval, and every quality probe above fits in about 2k — on safety behaviour, or with thinking enabled — which is how this model family is normally used. The requant rewrites the dense linears and lm_head, so those are exactly the places a regression could hide from these four probes. If you depend on any of them, measure it yourself before adopting this checkpoint; requant/verify.py takes a different prompt set with one flag.

For reference, the base release's own BF16-vs-NVFP4 card numbers (quoted from the base card at the pinned revision): GPQA Diamond 0.9217 -> 0.9211, SciCode 0.5621 -> 0.5769, MMMU Pro 0.7688 -> 0.7630, AA-LCR 0.7100 -> 0.7106, IFBench 0.6130 -> 0.6054, Terminal-Bench 2.1 0.8258 -> 0.8315. Those card figures come from standard public benchmarks under NVIDIA's own harness and are not comparable with the eval-200 numbers above, which are scored by this repository's own strict grader on Japanese prompts. Nothing here places my checkpoint on the same axis as the base card's benchmarks.

Speed (measured, with conditions)

Ruler: 64 Japanese prose prompts, temperature=0, max_tokens=512, thinking skipped via an empty assistant continuation. C=1 streams the prompts sequentially; tok/s = completion tokens / whole-window wall time including TTFT. Hardware: 2x DGX Spark (128 GB unified memory each), TP=2 over Ray, RoCE link (ConnectX-7 class) between the nodes. Serve flags in the repo's scripts: --max-model-len 204800, --max-num-seqs 20, --gpu-memory-utilization 0.85, --enable-prefix-caching, FP8 KV cache and RAY_memory_usage_threshold=0.99 on both nodes. The rows in the table ran the measurement rig instead — almost all of them at --max-model-len 16384 with --max-num-seqs 20 and --gpu-memory-utilization 0.86 and --enable-expert-parallel, except the 19.05 stock + MTP row which ran with expert parallel off (ep = off in the table), the pair-2 stock baseline at --max-model-len 131072 with 32 slots, and the pair-2 stock RDMA row at 16384 with 32 slots. Image: vllm/vllm-openai:glm53-flash-arm64-cu130 (+ derived RDMA image).

The shipped window is longer than the rig one on purpose. Re-measured on 2026-09-17 at the shipped window, with expert parallel on as everywhere else in this table, the ruler reads 34.78 tok/s — TPOT median 28.1 ms, TTFT median 0.313 s, 0 of the 64 requests failed — against 35.09 at 16384 / 20 / 0.86. That is a 0.9% difference across a 12.5x longer window, inside the run-to-run spread of about 2% on identical configurations. That 34.78 figure came from the old internal launcher (tools/dgx/llm/glm-serve.sh, expert parallel on), not from the serve scripts this repository ships — I keep it here only as the EP-on comparator for the two numbers below.

The serve scripts in the setup repo do not enable expert parallel, while every row in this table was measured with it on. Run exactly as they ship, the same ruler reads 37.33 tok/s (36.95 on a repeat); adding the two EP flags back drops it to 35.05, reproducing the 34.78 row to within 0.8%. EP is also hungrier on this hardware, so the scripts keep it off. All four of these numbers — 37.33, 36.95, 35.05, and 34.78 — were recorded with --max-num-seqs 20, and none of these figures is a speed guarantee under full load or in every environment. One operational note that matters more than either number: DGX Spark is unified memory, the engine sizes its budget from whatever was free when it profiled, and the same script has reached READY with anywhere from 5470 to 10270 MiB of host memory left. About 4.7 GiB of that is spent after READY, the first time each larger prefill shape arrives, so a boot that starts too low is killed later by one long prompt — and DGX OS ships no OOM daemon, which makes that a hung node rather than a killed process. The setup repo ships serve/check-headroom.sh as an optional diagnostic for checking this by hand; a low reading by itself does not refuse the boot. The longer window does not stop the server from booting with --max-num-seqs 20: it reaches READY at 204800 (909 s, then the same 0-failure ruler), which it did not at 307200. That is a boot-time acceptance fact, not a measurement of how many of those 20 admitted slots can actually run at once — the KV pool decides that, and at this window and around 25,000 tokens per prompt the number is 6: sending 20 requests at once queued 14 of them and ran 6 concurrently, with zero failures and zero preemptions even at 128% of KV pool capacity. --max-num-seqs 20 still ships unchanged — it sets what the scheduler will accept, not what it decodes together, and lowering it would not change the 6. Utilization is the one flag that moved down rather than up — 0.89 has been refused at boot on this pair, and at 0.88 the head node ran with about 2.1% of host RAM free — close enough to exhaustion that it does not matter what reaps the worker first — so the scripts ship 0.85 and leave 8.6%. The requant is part of why the window fits: at 0.88 the stock checkpoint tops out at 156672 tokens (vLLM prints that ceiling when it refuses to start), while route h boots at 204800.

† these rows ran C=1 over the first 8 prompts of the ruler, not all 64.

The ep column records --enable-expert-parallel state per row — the 19.05 stock + MTP row ran with it off. The gate is split per row: quality gate covers the checkpoint-level legs (degenerate / PPL / eval-200); TTFT gate is transport-dependent and was probed once on the route-h sockets serving, so RDMA rows read "verified on sockets; not re-run under RDMA" rather than a four-condition PASS on that transport. The weighted TPOT ms column is recorded directly in five shipped JSONs (the four release re-runs plus the route-h no-draft RDMA row); for nine more rows I computed the same quantity myself from each run's per-prompt records in the pre-sanitisation factory log (not part of this repository) and added the result to the row's shipped results/logs/ entry. The 2026-09-13 stock baseline has no per-prompt records anywhere I could find, so that row's cell stays empty rather than filled with its own plain-mean figure, which is a different statistic. Per-row measurement excerpts ship in the repo under results/logs/ (see the source_log column of results/results.tsv).

K sweep, what is and is not settled: on route h + RDMA, pair 1, K=1 measured 34.28 tok/s and K=3 measured 31.98, both already in the table above; K=2 remains the best of the values measured, not a proven optimum. The re-measurement question is answered: the same config on the second node pair read 34.47 tok/s, and an interleaved TCP/RDMA re-run on a fresh, non-overlapping 64-prompt set with prefix caching disabled read 34.48 and 35.12 tok/s over RDMA and 23.75 and 24.35 over TCP — within ~2% of the original 35.09 on the RDMA side. The headline run: 32407 completion tokens over 923.43 s of wall, 0 failed requests. C=32 aggregate on stock was 95.08 tok/s (sockets) and 109.03 tok/s (RDMA); both of those passes ran with 32 sequence slots, which the shipped --max-num-seqs 20 cannot reproduce, so read them as rig numbers rather than as what the released scripts do. No C=32 row exists for route h yet.

Long context, measured

The probe is the companion repo's bench/longctx.py: Japanese filler grown to a target length, ten facts planted at head, middle and tail token depths, one question each, temperature=0, graded by exact code match rather than by a judge. The set is bench/longctx-probe.jsonl, and every document is the shared prefix of its ten questions, so one long prefill is paid per length. One pass on the shipped configuration — --max-model-len 204800, --max-num-seqs 20, --gpu-memory-utilization 0.85, --enable-prefix-caching, route h with MTP at K=2 over RDMA:

40 of 40, no depth weaker than another: head 12/12, middle 16/16, tail 12/12. One reading from a different configuration, kept out of the table on purpose: at --max-model-len 307200 with utilization at 0.88 — not what the scripts ship — a single 204,767-token document answered 10/10. It needs a window the released scripts do not open, so it is not a row above.

This is retrieval, not quality. It says the model still finds a planted fact at 194,544 tokens; it says nothing about whether its prose or its reasoning hold up there.

Three limits come with the long window.

• A prompt the size of the window does not fit. The probe's top stage aims at 190k rather than 200k for a measured reason: a 204,754-token document plus max_tokens=64 is 204,818 against a 204,800 limit, and the server returns HTTP 400. The engine survives — the next request is answered normally — but that document cannot be asked about. The usable ceiling is the window minus the generation budget minus the chat template.
• The first token on a long prompt is slow. Prefill holds between 1620 and 1641 tok/s across all four lengths, so the 194,544-token prompt takes 120.04 s before the first token appears. Nothing is stuck; that is prefill. --enable-prefix-caching is in the shipped flags for this reason: pin one long document as the prefix and vary only the question, and every request after the first is an order of magnitude faster — at 130,990 tokens, 79.86 s cold against 3.15 s warm.
• Past about 259,000 tokens nothing gets through on this hardware. Not with a longer --max-model-len, not with a lower --gpu-memory-utilization, not with a smaller --max-num-batched-tokens: host memory runs out and the node's OOM reaper takes the vLLM worker. The peak activation figure vLLM prints at boot is measured on an 8192-token dummy run, and the scratch space the sparse-attention indexer wants during a long prefill sits outside the --gpu-memory-utilization budget entirely. Dropping utilization from 0.88 to 0.85 moved the death from 62 s into the prefill to 155 s; it did not make the prompt fit.

RAY_memory_usage_threshold=0.99 is exported on both nodes by the repo's serve/start-head.sh and serve/start-worker.sh, and it is not optional. Unified memory means --gpu-memory-utilization is taken out of host RAM, so Ray's own OOM monitor sees the node above its default 0.95 and kills the largest actor it can find, which is the vLLM TP0 worker. What you see is EngineDeadError — at boot, or in the middle of a request — with nothing wrong in the vLLM log; the kill is in the raylet log. The two scripts bring up two separate raylets, each with its own monitor, so the variable has to be exported on both.

Serving

The served API has no authentication, and it listens on every interface. The scripts below pass --host 0.0.0.0, so anything that can reach port 8000 on either node can send requests. Keep this on a network you trust and do not expose it to a public one; access control (firewall rules or the equivalent for your site) is yours to arrange, and if the endpoint has to be reachable more widely, put authentication and TLS in front of it. Ray needs separate treatment: its dashboard and inter-node ports have to be isolated on their own, because authentication in front of the API does not protect Ray. A long enough prompt can also exhaust host memory on this hardware (see "Long context, measured"), so an endpoint left open is worse here than the usual.

2 DGX Spark nodes, TP=2. Summary of the repository's pipeline:

# on each node: pull the official image, then build the RDMA derivative
# (installs Ubuntu questing rdma-core over the noble base; self-verifies
# the MLX5_1.25 symbols the NCCL NET/IB plugin needs)
docker pull vllm/vllm-openai:glm53-flash-arm64-cu130
docker build -t vllm/vllm-openai:glm53-flash-arm64-cu130-nccl-ib \
    - < docker/Dockerfile.nccl-ib

# serve.env filled per node, then:
serve/start-head.sh     # Ray head + vllm serve
serve/start-worker.sh   # joins the Ray cluster

The headline speed is a draft number — the + MTP rows above. If you bring the server up from the companion repository's serve/serve.env.example, set MTP_DIR there: the line is commented out by default, so an untouched file starts the server with no draft. Without the draft, the 64-prompt ruler reads 28.28 tok/s against 34.99 tok/s with the draft at K=2 — the no-draft figure sits alongside the 27.15 route-h no-draft row above. The draft directory is the one built above with requant/build-mtp-draft.py.

The effective vllm serve invocation (inside the derived image, with the kda-quant / mla-quant / mtp-bf16 overlays bind-mounted, and /checkpoint-mtp built as described above):

vllm serve /checkpoint \
  --served-model-name GLM-5.3-Flash-NVFP4-h \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 2 --data-parallel-size 1 \
  --distributed-executor-backend ray \
  --reasoning-parser glm45 \
  --kernel-config '{"enable_flashinfer_autotune": false}' \
  --kv-cache-dtype fp8 \
  --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 32}' \
  --max-num-batched-tokens 8192 --enable-chunked-prefill \
  --max-num-seqs 20 --max-model-len 204800 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --limit-mm-per-prompt '{"image":0,"video":0}' \
  --speculative-config '{"method":"mtp","model":"/checkpoint-mtp","num_speculative_tokens":2}'

NCCL_IB=0 in the repo's serve config falls back to the stock image over sockets. Do not raise num_speculative_tokens past 2 for this checkpoint at these memory settings (128 GB unified): every deeper K that was actually measured came out slower. On route h, K=3 measured 31.98 tok/s against K=2's 35.09; on the stock checkpoint K=4 gave 15.2 and K=5 gave 13.1 tok/s, below its own K=2 row — pair 2, sockets, expert parallel on, 64 prompts. Those two stock arms refused to start at the default CUDA-graph capture sizes and booted only once the capture sizes were narrowed to the ones the measurement needs. On route h, K=4 did boot (READY 918 s) but the arm was stopped by the operator before it measured, so no route-h number exists at K=4, and K=5 was never run on route h at all. This K=2 comparator (17.71 tok/s) is the stock row already in the table above; K=4 and K=5 stay here rather than in that table because they are boot/speed notes on the stock checkpoint, not on route h.

The model id a client has to send is GLM-5.3-Flash-NVFP4-h — the --served-model-name value above, not the checkpoint path; any other id comes back 404. That check is not theoretical: on 2026-09-17 a client still pointed at the earlier name, GLM-5.3-Flash-NVFP4, got 404 on every request until it was re-pointed at GLM-5.3-Flash-NVFP4-h. Confirm the served name before sending traffic:

curl -s http://127.0.0.1:8000/v1/models

An OpenAI-compatible client needs three settings: base URL http://127.0.0.1:8000/v1 (the head's address in place of localhost when the client runs elsewhere), model GLM-5.3-Flash-NVFP4-h, and any string as the API key — the server does not check one. I drive this server from a coding agent set up exactly that way: pi, pointed at base_url http://<head-node-ip>:<PORT>/v1 with model GLM-5.3-Flash-NVFP4-h, gets thinking back in its own field instead of it leaking into the reply.

Read reasoning as well as content. --reasoning-parser glm45 moves the model's thinking into a separate reasoning field on the message and leaves content holding the answer. chat_template_kwargs: {"enable_thinking": false} stops the parser rather than the model, so the thinking lands in content instead of disappearing — send neither and read both fields.

Do not let the window fill. "Long context, measured" above records four prompts that killed the engine outright regardless of --gpu-memory-utilization or --max-num-batched-tokens; that failure mode is not one you can prompt-engineer around once the window is nearly full. Treat compaction as a routine part of the client, not a reaction to a failure: fold or clear an agent's conversation on a schedule well short of the window, before the server is forced to do it for you.

<!-- SAFE-WINDOW-TBD --> marks where the exact safe threshold belongs

once it is measured; it is not measured yet, so this repo does not publish a number it has not tested — build in margin instead of trusting one.

How many requests actually run at once is not the flag value. The invocation's --max-num-seqs 20 is a ceiling the scheduler is allowed to admit, not a guarantee that 20 requests run together; when the KV cache pool cannot hold that many sequences, the scheduler queues the rest rather than failing the request or preempting a running one. Sending 20 requests at once, each about 25,000 tokens, against this pair's 382,740-token KV pool (204,800-token window, expert parallel on) queued cleanly at 128% of pool capacity: the scheduler ran a peak of 6 requests concurrently, held the other 14 in the waiting queue, and finished all 20 with zero failures and zero preemptions. So --max-num-seqs 20 is the number of requests the server will accept, not the number it runs together — the number this configuration actually decodes at once, at this prompt length, is 6 — and there is no reason to lower the flag.

Serving script additions, 2026-09-19 (measured)

A minor update on v1 adds two optional environment variables to the serving scripts: NCCL_NCHANNELS (to reduce NCCL channel count in RDMA mode) and JIT_CACHE_DIR (to persist compiled kernel caches across restarts). If both are unset, behavior is identical to the previous version.

Measured in this configuration (two-node pair, as described above):

• Host memory at READY: +3,455 MiB on the first node pair; on a second pair, +1.5–2.1 GiB (direction reproduced, magnitude did not match).
• Startup time with persistent JIT cache: 582 s vs. 684 s (cold compile).
• Total throughput (tok/s) at 1/2/4/8 concurrent requests: 28.7/36.9/45.4/55.9 (NCCL 8) vs. 29.0/36.8/44.9/54.9 (NCCL 64); no speed penalty found under the measured conditions.
• Tested to 197,486 tokens in ascending order on a single engine instance; not claimed as a general property of input length.

Not measured: sustained production traffic, all input lengths under high concurrency, or the consistency of memory gains across different hardware.

For detailed measurements, conditions, and methodology, see the serving notes report.

What is left for a v2

This is a list of what I know is unfinished, not a roadmap. I am not predicting how much any of it is worth; I will publish a number when I have measured one.

Why boots and long prompts still fail — four separate causes, not one

• Boot-time host headroom varies. The same script has reached READY with anywhere from 5,470 to 10,270 MiB of host memory free (see "Speed, measured" above); a boot that starts low can be killed later by a single long prompt.
• Ray's OOM monitor. Without RAY_memory_usage_threshold=0.99 exported on both nodes, Ray's own default of 0.95 sees the node over budget first and kills the vLLM TP0 worker — already worked around, but the export is required on both nodes, not optional.
• Demand past the window. Prompts around 259,000 tokens sit outside the current window on this hardware regardless of --gpu-memory-utilization or --max-num-batched-tokens; the top edge of the window is still not cleared.
• Failures that are not about memory at all. --max-num-batched-tokens 2048 hits an illegal-memory-access in the KDA kernel even with headroom to spare, and an nccl-ll-symm-ar build returned empty responses instead of failing loudly. Both are excluded from what ships here.

serve/check-headroom.sh's boot floor is a provisional screening value pulled from the EP-off ladder above, not a safety guarantee, and it does not imply any particular pass rate; I am not putting a probability figure on how safe a passing boot is. It is an optional diagnostic I run by hand, not a required gate; screening a low boot out at start-up is a screen, not a fix for the causes above.

Speed, still open

• My own NVFP4 MoE kernel. It has never actually run in a served request: three attempts hooked a class the DGX Spark build does not instantiate, then the wrong class, then a host-side call that threw and silently disabled the overlay. Until it runs with a proof-of-life line in the engine log, there is nothing to measure.
• All-reduce. Cross-node peer wait was 20.8 ms of an ~87 ms decode step in the one step attribution I ran (eager, stock, single stream). RDMA cut the waiting a lot; what remains has not been re-attributed under RDMA, and I have not tried to reduce the number of collectives itself.
• A trained draft model. The checkpoint's own MTP head accepts 0.62 of the tokens it proposes. A draft trained on this model's own outputs could accept more, which shortens every cycle. My first attempt is data-starved — 594 training records, about 301k tokens against a ~3M-token setup target, holdout per-slot hit 0.03-0.08 — and needs roughly ten times the data before it is worth measuring.
• Speculation depth. K=1 (34.28) and K=2 (35.09) are not separated by this experiment's run-to-run spread, and K=3 is slower. A better draft would change where that optimum sits.

Quality, not yet measured

• English and other languages. Every probe here is Japanese.
• Code correctness. Code prompts appear in the speed table and nowhere in the quality table.
• Multi-turn conversations, tool calling, instruction following, long-context quality — the needle probe reaches 194,544 tokens at 40/40, but that is retrieval and nothing in the quality table ran on a prompt longer than about 2k — safety behaviour, and the model with thinking enabled, which is how this family is normally used.
• Perplexity on a larger corpus, with a confidence interval. The current probe is eight sentences and the code does not keep the per-token values, so the ratio has no interval attached to it.
• A paired test on eval-200. The grader discards per-item results, so the same 150 items cannot be tested as pairs, which is what they are.

What is still missing from the reproduction path, and the list of things I have already measured as not working (FP8 dense linears, K of 3 or more, forced eager mode), are in the companion repository: its clean-room section and its failure table.

Versions

This is the v1 drop (git tag v1): the route-h checkpoint and the measurements above. If you only need the weights, v1 is what you want. A v2 would appear on this same repo only if a faster configuration — a trained drafter, or a better K from a holdout sweep — clears the same quality gate; nothing of it is published yet.

2026-09-19, minor update on v1: two optional serving-script env vars (NCCL_NCHANNELS, JIT_CACHE_DIR); checkpoint unchanged. See Serving script additions above.

Config expectations

The requantization rewrites the quantization metadata for the modelopt MIXED_PRECISION loader:

• config.json: quantization_config.quant_algo = MIXED_PRECISION; quantization_config.producer.requant = requant.py (weight-only NVFP4 quantization); producer.requant_target = h; quantized_layers holds 369 modules, each with quant_algo + group_size = 16 — 318 at W4A16_NVFP4 (what this pass converted) and 51 left at the stock NVFP4 (the 42 routed expert blocks plus the 9 dense-MLP projections of layers 0-2). ignore has 45 entries: the 42 per-layer mlp.gate routers, embed_tokens, model.visual*, and model.language_model.layers.45* (the MTP layer stays BF16 and must remain excluded).
• hf_quant_config.json: quantization.quant_algo = MIXED_PRECISION, kv_cache_quant_algo = FP8, group_size = 16, the same 369-entry quantized_layers and the matching 45-entry exclude_modules.
• model.safetensors.index.json: weight_map repointed at the 18 regenerated shards and covering 148,659 tensors; metadata.total_size = 193,304,829,584.

The repo's requant/verify.py config <dir> statically gates all of this (missing NVFP4 global scales show up as KeyError ...weight_scale_2 at load — the gate catches them first).

Files in this repo

The weights ship as 18 safetensors shards, model-00001-of-00018.safetensors through model-00018-of-00018.safetensors. Most of them are about 10.3 GiB; the set is 193,324,988,184 bytes on disk, holding 148,659 tensors.

License and attribution

MIT. LICENSE carries the upstream MIT text (Copyright (c) 2026 Z.AI Co., Ltd) and NOTICE records the attribution chain. Model by Z.AI (zai-org/GLM-5.3-Flash); base NVFP4 quantization by NVIDIA with Model Optimizer v0.47.0; the route-h quantization is the modification published here. Ready for commercial and non-commercial use under the same terms as the base release.

Contributors

Contributors: tenhkspark; GLM-5.3 and GLM-5.3-Flash (Z.ai) assisted with analysis, test sets and documentation.