Kwaipilot-KAT-Coder-V2.5-Dev-NVFP4-MIXED

KAT-Coder-V2.5-Dev — NVFP4 for veloGB10 (NVIDIA DGX Spark / GB10)

Original model: Kwaipilot/KAT-Coder-V2.5-Dev by the KwaiKAT team, Apache-2.0. This artifact contains modified weights (NVFP4/FP8 quantization); the original model is © its authors.

NVFP4/FP8-quantized variant of KAT-Coder-V2.5-Dev. This quantized artifact exists first and foremost to run with the veloGB10 inference engine (gb10_inference) on NVIDIA DGX Spark / GB10 systems — it is produced, validated, and tuned for veloGB10's kernels. The weights use the standard HuggingFace compressed-tensors layout (nvfp4-pack-quantized), so this artifact can be used for any purpose, with any framework that reads the format. It is in particular fit and proven to work with veloGB10 — greedy-lossless speculative decoding (where applicable), two-node TP=2 serving, and the engine's full correctness test suite all pass on this artifact.

Specifications

Running with veloGB10

Full instructions: veloGB10 README (build or download a release binary, then). The binary loads its PTX kernel artifacts from the current directory — run it from wherever the binary + *.ptx live.

Single user — maximum speed:

gb10_inference --server --model-dir=/path/to/Kwaipilot-KAT-Coder-V2.5-Dev-NVFP4-MIXED --port=9000 \
  --max-seq-len=32768 --max-batch=1 --max-tokens=4096 \
  --prefix-cache=on --mtp=off --default-presence-penalty 1.5

Four concurrent users — maximum aggregate throughput:

gb10_inference --server --model-dir=/path/to/Kwaipilot-KAT-Coder-V2.5-Dev-NVFP4-MIXED --port=9000 \
  --max-seq-len=32768 --max-batch=4 --max-tokens=4096 \
  --prefix-cache=on --default-presence-penalty 1.5

Two-node TP=2 (maximum speed and capacity):

On the second node (no model copy, no configuration — the head ships weights, config, and calibration at sync):

gb10_inference --node --port 29500

On the head node:

GB10_TP_SHARD_MIXERS=1 gb10_inference --server --model-dir=/path/to/Kwaipilot-KAT-Coder-V2.5-Dev-NVFP4-MIXED \
  --tp --nodes <peer-ip>:29500 --port 9000 \
  --max-seq-len=32768 --max-batch=1 --max-tokens=4096 \
  --prefix-cache=on --default-presence-penalty 1.5

Measured on veloGB10 (greedy, NVFP4)

Pending — single-node and TP=2 throughput numbers are being gathered and will be added here once the full gate suite completes. See the veloGB10 README for the latest measured tables across the supported model family.

License & credits

• Original model: Kwaipilot/KAT-Coder-V2.5-Dev — © KwaiKAT team, Apache License 2.0.
• This artifact: quantized from the original with veloGB10's offline quantizer (nvfp4-mixed). Weights are modified relative to the original; tokenizer and chat template are unchanged from the original model.
• Engine: veloGB10 (Apache-2.0).