GLM-5.3-Flash-NVFP4

GLM-5.3-Flash NVFP4

NVFP4 (4-bit e2m1 weights with an FP8-e4m3 per-16 scale plane, 4.5 bits/element) weight-only mint of zai-org/GLM-5.3-Flash, produced with NVIDIA TensorRT Model Optimizer 0.46.0 and gated against an unfused f32 reference executor before publication.

Built and validated with the tiyuvta serving engine on RTX Blackwell, by tiyuvta (inference.tiyuvta.ai).

This is a bring-up artifact, not a hosted offering. We do not serve GLM-5.3-Flash to customers, there is no endpoint and no model id for it, and every number below is a dated bench measurement on named hardware rather than a product claim. The line above says who made this and what ran it.

• 190.7 GB across 20 safetensors shards, 38,770 tensors: 37,338 quantized, 1,432 kept.
• Source is the vendor's BF16 twin, not their FP8 release. Never quantize from a quant when the full-precision twin ships.
• Upstream technical report: GLM-5: from Vibe Coding to Agentic Engineering (arXiv 2602.15763).

How it was made

Per tensor W [out, in]:

weight_scale_2 (f32 scalar)      = amax(|W|) / (6 * 448)
weight_scale   (e4m3, [out, in/16]) = block_amax / (6 * weight_scale_2), zeros -> 1.0
weight         (u8,   [out, in/2])  = e2m1 codes, element 2i in the low nibble

One trap worth repeating: NVFP4QTensor.quantize(..., try_tensorrt=True) on a box with tensorrt_llm installed hands back CUTLASS-swizzled scales rather than the modelopt layout. This mint uses the default try_tensorrt=False and asserts the scale shape and dtype on every tensor.

Precision split

GLM-5.3-Flash is a hybrid stack: 45 decoder layers plus one MTP layer, 34 of them KDA linear-attention and 11 DSA (MLA with a sparse indexer), MoE on 42 layers with 288 routed experts plus one shared, and 3 dense layers. The split mirrors the vendor's own FP8 exclusions and the tensor census, not a guess about which layers "look big".

The keep list ships in both dialects in config.json: modules_to_not_convert and compressed-tensors ignore, 628 entries each. That redundancy is deliberate. Writing it in only one dialect made it invisible to a loader that reads the other, which silently re-encoded the large KDA projections to 8-bit against the intent of the split. Stating your own fact in the dialect the reader speaks is cheaper than teaching every reader a new dialect.

The gate this passed

Same engine (the serving engine's unfused f32 streaming reference executor), same tokens, full last-position vocabulary row of 154,880 logits, all three artifacts run on one box:

The middle row is the point. An absolute logit delta has no interpretation without a same-instrument reference point, and a 4-bit mint compared against an 8-bit artifact measures two error sources blended together rather than one. Compared against its own full-precision source, this mint's deviation is comparable to the vendor's own 8-bit quantization deviation, at half the bit width.

The gate was re-run on the exact bytes published here, not inherited from a run over a directory with the same name. Publishing a mint that has only a family-level gate is how you ship a quantization that scrambles rankings while every smoke test stays green.

What this gate does not claim: serving accuracy over long generations, long-context behaviour, sampled decoding quality, or any engine's fused quant kernels. The reference runner dequantizes to f32; a serving path riding 4-bit matmul kernels is a separate gate.

Why bring-up found things, and what it found

Every defect in this bring-up was caught by an instrument built to disagree loudly: the serving engine's unfused f32 reference executor run against the serving path on the same input, byte oracles over rendered prompts and tokenizer output, and refusals written to fire rather than to cope. Not one was caught by the engine looking unhealthy. A wrong forward pass on a large language model does not crash and does not obviously degrade. It answers fluently, at full speed, off distribution, and that is why the instrument has to be a second implementation rather than a closer look at the output.

Twelve of them landed in this lane, so here they are rather than as a number you have to take on faith:

That is thirteen rows. Twelve of them changed what the model produced; row 1 refused loudly instead, which is the outcome you want and the reason it is listed separately.

The bring-up log these come from (glm53-flash-bringup-20260827, banked in the lab's research record) records each defect with its before and after logit rows, alongside the mint receipts and the gate coverage for the exact bytes in this repo.

Two are worth expanding, because they generalize past this model:

Two missing rows in a name map. The engine's ggml-to-HF name map lacked entries for the router's e_score_correction_bias (nested under the router module, not where the DeepSeek-V3 lineage puts it) and for the shared expert's ffn_{gate,up,down}_shexp weights (the generic map spells the shared expert singular; this checkpoint spells it plural). Effect: the shared-expert branch was silently dropped from every MoE layer, and router selection drifted. The tensor census passed the whole time, because a census exercises the contract's names, and the contract's names were right. The property no existing gate covered was name resolution on the engine's own loading path.

The chat template is not ChatML, and a ChatML lookalike returns 200. Serving this model through a ChatML renderer produced fluent text at every surface: chat completions, streaming, the Anthropic and OpenAI translation surfaces, and tool calls. All of it ran off distribution. The <|im_start|> and <|im_end|> frame is not in this checkpoint's special vocabulary at all, so it tokenized as ordinary text. Tool calls only parsed because the model obeyed the foreign instruction block it was handed; the native wire had no parser branch and would have surfaced verbatim as content. reasoning_effort was accepted and ignored, provable because a request with effort: "low" hit the prefix cache at 288 of 288 tokens against the identical request without it: zero rendered bytes differed.

The native dialect, for anyone else serving this checkpoint:

• [gMASK]<sop> framing, with <|system|> / <|user|> / <|assistant|> / <|observation|> role markers.
• Tool calls: <tool_call>NAME<arg_key>city</arg_key><arg_value>Paris</arg_value></tool_call>.
• Tool results come back in an <|observation|> block, with run grouping and id-keyed re-ordering.
• reasoning_effort has a real rung above high. A clamp written for a three-rung family eats it.

Read the template out of chat_template.jinja in this repo and render through it. Do not pattern-match a template to a family you already support: the failure mode is a 200 with good prose.

Running it

This artifact runs on the tiyuvta serving engine. It is a modelopt-quantized glm5_next checkpoint, so a stock transformers load is not the path, which is why library_name here says safetensors rather than transformers: a transformers usage snippet on this repo would be a button that does not work.

190.7 GB does not fit a single 96 GB card, so the routed experts stream from host memory through an SLRU residency cache while the trunk stays resident. Pinned-DMA staging over the pageable path is worth roughly 2.4x on this model.

Measured decode, so the offload cost is not a mystery

Bench measurement, 2026-08-28, on one RTX PRO 6000 Blackwell Server Edition (96 GB, 500 W) of a two-card box, card 0 only, single stream, greedy, NVIDIA_TF32_OVERRIDE=0, 12,000 expert slots, trunk resident at 79.6 GB:

20.3 tok/s, median of 3 reps, reproduced across six boots.

Greedy is used here because it is byte-deterministic and therefore checkable, not because it is how anyone should run the model. Where the time goes, solved from measurement rather than modelled: about 33 ms per token of resident work and the rest expert staging over PCIe, at roughly 53 GB/s on the pinned path against about 10 GB/s pageable. The resident-traffic roofline for this configuration is 63 tok/s, so this number has room in it and is not a property of the artifact. It is a bring-up figure on one configuration.

Context: 1,035,357 tokens demonstrated, and where the remaining wall actually is

The upstream checkpoint declares max_position_embeddings: 1048576. We have run this artifact at that scale, once, deliberately, and the receipts are dated bench measurements on named hardware: not a hosted offering, and not a latency claim.

Demonstrated (2026-08-29, 4x RTX PRO 6000 Blackwell 96 GB, PP4 pipeline on the serving engine, stage splits at 13, 26 and 39): a real 1,035,357-token prompt (Gutenberg prose, sha-banked, token count from the server's own usage, cached_tokens=0) primed through the serving surface inside the model's 1,048,576 window, then decoded greedy to EOS with a coherent cross-book answer. Prefill 161.28 tok/s (6,419.8 s wall, 107 minutes); the vendor-default sampled twin primed the same prompt at 161.26 tok/s (two independent full primes agreeing to 0.01%) and also answered coherently. Error census of the serve log: 0.

Depth behaves, with no cliff:

Chunked prefill throughput is depth-flat to 1M (-6.6% from 16k), and greedy decode decays smoothly (1.75x over three orders of magnitude of depth); the sampled twin sits within 0.6% of greedy at 1M. In-context retrieval held at depth (at 131k the answer cited a detail from a salon debate inside War and Peace).

What the earlier revision of this card called the wall, the DSA score-plane transient of a monolithic prefill (N^2 bytes per MLA layer per call), is fixed, not by more VRAM but by the chunk schedule it predicted: the serving engine's mHC prime now walks bounded chunks (the monolithic path is preserved as a rollback), so the per-call plane is chunk_rows x (N / 4) f32 rather than N^2 bytes. That was never a quantization artifact and its fix is not one either: an FP8 or BF16 copy of this model has the same arithmetic on both sides of the fix.

Honest boundaries, so this is not read as a serving claim:

• 1M is demonstrated on 4 cards, not fewer. A 3-card full-expert-residency configuration of this artifact fails the same prime (GPU out-of-memory in the DSA k-pool selection at a 97.2 GiB per-card peak), so the depth ceiling of smaller placements sits well below 1M, and the only demonstrated 1M configuration is the 4-card pipeline with the expert cache capped to leave the tail stage room.
• Time-to-first-token at 1M is 107 minutes on that baseline. Long-context capacity is solved; long-context latency is engineering in progress (prefill has since moved independently on other configurations of the same engine, e.g. a tensor-core MLA prefill path measured at 1629-2255 tok/s at short depths, but no 1M number exists on that path and none is claimed here).
• Everything above is greedy-plus-sampled-twin bench measurement on named hardware, with its receipts banked in the lab's research record (glm53-flash-bringup-20260827/1m-demo-20260829).

Files

The vision tower is present in the source architecture and is not exercised by this lane's gates. Text is what was gated.

Attribution and licence

MIT, following upstream zai-org/GLM-5.3-Flash. All model capability belongs to the GLM-5 team; this repository contributes a quantization and the receipts that it is faithful to its source.

@misc{glm5team2026glm5vibecodingagentic,
      title={GLM-5: from Vibe Coding to Agentic Engineering},
      author={GLM-5-Team},
      year={2026},
      eprint={2602.15763},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2602.15763},
}