KAT-Coder-V2.5-Dev-gguf

KAT-Coder-V2.5-Dev — 11.7 GB Asymmetric GGUF for QuarkStar

These GGUF weights are built for QuarkStar, a small native inference engine specialized for Qwen3.6-35B-A3B.

This repository contains an imatrix-calibrated GGUF quantization of Kwaipilot/KAT-Coder-V2.5-Dev, a 35B-A3B Mixture-of-Experts model focused on agentic coding and repository-level software engineering.

KAT-Coder-V2.5-Dev is post-trained from Qwen/Qwen3.6-35B-A3B. It is designed for tasks such as:

• understanding unfamiliar repositories;
• locating and fixing bugs;
• editing multiple files;
• using shell, search and file tools;
• running and interpreting tests;
• implementing features from issue descriptions;
• long-horizon coding-agent workflows.

The quantization recipe follows the asymmetric MoE strategy explored by Salvatore Sanfilippo in DS4: aggressively compress the routed experts, which contain most of the model weights, while retaining substantially higher precision for the attention path, shared experts and output tensors.

The resulting model is approximately 11.7 GB, making a 35B-class coding model practical on machines with 16 GB of unified memory.

This is a quantized derivative. Refer to the upstream KAT-Coder model card and technical report for training details, intended use, benchmark methodology and model limitations.

Model details

The upstream context limit is an architectural maximum. Practical context size depends on available memory, KV-cache precision, runtime buffers and the inference engine.

Files

SHA-256:

13e3f7c3f9fc3b6dc85b63e9737dce9ea049f2759db0fb269b0fc43b1b7c0a90

This repository currently contains one language-model GGUF. It does not include a separate multimodal projector or MTP speculative-decoding model.

Quantization recipe

Unlike a uniform 2-bit quantization, this model applies low precision primarily to the routed MoE experts.

Conceptually:

routed gate/up experts -> IQ2_XXS
routed down experts    -> Q2_K
attention projections  -> Q8_0
shared experts         -> Q8_0
output head            -> Q8_0

The routed experts account for most of the model's storage footprint. Compressing them aggressively provides most of the size reduction while preserving higher precision in components that directly control attention, routing and token prediction.

The imatrix calibration data is used to make the low-bit expert quantization more sensitive to tensor importance than a purely weight-only conversion.

Why this quantization?

A conventional Q8 GGUF of this model is roughly 37 GB, while the original BF16 weights are roughly 69 GB.

This asymmetric release targets a different operating point:

• a fully local 35B-class coding model;
• approximately 11.7 GB of model weights;
• enough remaining memory on a 16 GB unified-memory system for runtime buffers, the operating system and a moderate KV cache;
• higher precision for attention, shared experts and output generation;
• aggressive compression concentrated in the routed experts.

This is especially useful for Mixture-of-Experts models because only a subset of the routed experts is activated for each token.

QuarkStar / q36

This GGUF targets the Qwen3.6-35B-A3B-compatible tensor layout used by QuarkStar.

QuarkStar is a small native inference engine with Vulkan and Metal backends. It is deliberately specialized rather than being a general-purpose GGUF runner.

Vulkan

git clone https://github.com/Ninnix/q36
cd q36

make
mkdir -p gguf

./download_model.sh kat-coder

./q36 \
  -m gguf/KAT-Coder-V2.5-Dev-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf \
  --ctx 32768 \
  -p "Inspect this repository and explain its architecture."

Apple Silicon / Metal

make metal

./q36 \
  --metal \
  -m gguf/KAT-Coder-V2.5-Dev-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf \
  --ctx 32768 \
  -p "Write a minimal Redis-compatible server in C."

OpenAI-compatible server

./q36-server \
  -m gguf/KAT-Coder-V2.5-Dev-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf \
  --ctx 32768

QuarkStar changes quickly and is intentionally model-specific. Use a revision that supports the KAT-Coder GGUF and its embedded metadata.

Upstream model

• Model: Kwaipilot/KAT-Coder-V2.5-Dev
• Technical report: KAT-Coder-V2.5 Technical Report
• Foundation model: Qwen/Qwen3.6-35B-A3B

KAT-Coder-V2.5 is trained for autonomous work inside executable repositories rather than only single-turn code generation. The upstream project combines supervised training, reinforcement learning, executable environments, verifiable rewards and multi-teacher on-policy distillation.

Acknowledgements

The asymmetric expert-quantization strategy and runtime direction were inspired by Salvatore Sanfilippo's work on:

• DwarfStar

This release also depends on the formats, quantization methods, kernels, tests and engineering knowledge developed by:

• Kwaipilot
• Qwen
• llama.cpp
• GGML
• Salvatore Sanfilippo
• Georgi Gerganov
• llama.cpp and GGML contributors

License

The upstream KAT-Coder-V2.5-Dev model, its Qwen foundation and these derived GGUF weights are distributed under the Apache License 2.0.

The QuarkStar / q36 inference-engine source code is distributed separately under the MIT License.

See the upstream repositories for their complete license text, acceptable-use guidance and attribution requirements:

• Kwaipilot/KAT-Coder-V2.5-Dev
• Qwen/Qwen3.6-35B-A3B