KAT-Coder-V2.5-Dev-VL-Flash

KAT-Coder-V2.5-Dev-VL-Flash

NVFP4 build of beyoru/KAT-Coder-V2.5-Dev-VL, which is itself the vision-enabled build of Kwaipilot/KAT-Coder-V2.5-Dev.

⚠️ Requires an NVIDIA Blackwell GPU

NVFP4 runs 4-bit weights and 4-bit activations on Blackwell FP4 tensor cores. This checkpoint needs compute capability sm100 / sm120: B200, B300, RTX PRO 6000 Blackwell, RTX 50-series, DGX Spark. It will not run on Hopper (H100/H200), Ada (L40S/RTX 40), or Ampere (A100). On those GPUs use the bf16 build beyoru/KAT-Coder-V2.5-Dev-VL instead.

⚠️ Required flag: --max-num-batched-tokens 8192

This is a hybrid attention + Gated-DeltaNet architecture. With prefix caching on (the default in recent vLLM), vLLM forces mamba_cache_mode='align' and then raises the attention block size to 2096 so that "attention page size >= mamba page size". The default max_num_batched_tokens is 2048, so engine startup dies with:

AssertionError: In Mamba cache align mode,
                block_size (2096) must be <= max_num_batched_tokens (2048)

Any value >= 2096 fixes it; 8192 also helps prefill. (Disabling prefix caching works too — mamba cache falls back to mode none — but you lose the prefix cache, which is not worth it.) This applies to the bf16 parent as well.

Loading status on Blackwell

Confirmed loading and initialising on sm121 with vLLM (nightly 0.1.dev18498):

Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend
Loading weights took 114.37 s -> 20.61 GiB
FLASH_ATTN for vit attention / MMEncoderAttention     <- vision tower initialises
torch.compile 36.83 s · CUDA graphs PIECEWISE=51 FULL=35 · KV cache 83.33 GiB

Weights, NVFP4 kernel selection, the MoE backend, the vision tower, compilation and CUDA-graph capture all come up cleanly. Generation quality on FP4 tensor cores has still not been benchmarked — no accuracy numbers are claimed here. Reports in the Community tab are welcome.

---

Precision layout

Mixed precision — not everything is FP4. Weight-only 4-bit (W4A16) does not touch FP4 tensor cores, so the parts that dominate decode are W4A4, while the parts that hurt accuracy most stay higher.

Pushing every expert layer to W4A4 is the "Fast" trade: maximum FP4 tensor-core coverage, at some accuracy cost versus keeping the last layers in FP8.

---

Ready-made config

docker-compose.vllm.yaml — vLLM on a single Blackwell GPU, vision enabled, MoE backend left to auto-selection. Untested on real FP4 hardware (see the warning above).

SGLang users: FlashQLA's GDN prefill kernel applies to this architecture (30 of 40 layers are linear_attention) and is opt-in via --linear-attn-prefill-backend flashqla. Windowed-MTP does not apply — mtp_num_hidden_layers = 0, so there is no MTP/NEXTN head to self-speculate with.

Serving

vLLM

uv pip install vllm --torch-backend=auto      # vllm >= 0.26.0

vllm serve beyoru/KAT-Coder-V2.5-Dev-VL-Flash \
    --served-model-name kat-vl-flash \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --trust-remote-code

Let vLLM pick the MoE backend. Do not pin one. On Blackwell the CUTLASS / FlashInfer-TRTLLM / CuTe-DSL NVFP4 paths are the fast ones; forcing Marlin gives a large throughput regression because Marlin is a weight-only (W4A16) kernel and never reaches the FP4 tensor cores.

Do not pass --language-model-only — that flag belongs to the upstream text-only release and would disable the vision tower this build contains.

Transformers

Loading works (the quantization format is compressed-tensors), but transformers has no fast FP4 execution path — it dequantizes. Use vLLM or SGLang for anything performance-sensitive.

---

Usage

OpenAI-compatible API. This is a thinking model: it emits a reasoning block ending in </think>, so pass a generous max_tokens (≥256) and strip everything up to </think>, or disable thinking via chat_template_kwargs: {"enable_thinking": false}.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

r = client.chat.completions.create(
    model="kat-vl-flash",
    messages=[{"role": "user", "content": "Write a Python function to reverse a linked list."}],
    max_tokens=512, temperature=0.0,
)
print(r.choices[0].message.content.split("</think>")[-1].strip())

Sampling defaults follow upstream: temperature=1.0, top_p=0.95, top_k=20 for open-ended work, temperature=0.0 for deterministic coding.

---

Vision capability — READ THIS

The vision tower was never co-trained with this checkpoint's language model, which was heavily RL-tuned for agentic coding. Measured on the bf16 parent (this build inherits the behaviour, and 4-bit experts can only make it worse):

• ✅ Coarse visual questions: colour, shape, layout, rough scene gist.
• ❌ Not usable for: reading code from screenshots, OCR, UI labels, document or chart understanding — anything where one wrong character changes the answer.

If your workload is text-only, the vision tower costs you 0.9 GB and nothing else; the language model is what this checkpoint is for.

---

Acknowledgements

• Kwaipilot/KAT-Coder-V2.5-Dev — base model and all language-model weights (Apache-2.0).
• Qwen/Qwen3.6-35B-A3B — the architecture this model derives from (Apache-2.0).
• The NVFP4 mixed-precision layout follows the approach popularised by Unsloth's Dynamic NVFP4 quants: keep the accuracy-critical layers at FP8/BF16, push the rest to W4A4 so Blackwell's FP4 tensor cores are actually used.

Released under Apache-2.0, matching both upstream models.